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FROM  THE 
CHAIRMAN  AND 
EXECUTIVE 
EDITOR 

Dr.  Larrie  D.  Ferreiro 


The  theme  for  this  edition  of  Defense 
Acquisition  Research  Journal  is  “Tran¬ 
sitioning  to  the  Future.”  The  first  article, 
“Application  of  System  and  Integration 
Readiness  Levels  to  Department  of 
Defense  Research  and  Development”  by 
Sean  Ross,  demonstrates  howto  move 
beyond  the  Technology  Readiness  Level 
system  of  estimating  technological  matu¬ 
rity,  which  was  developed  by  NASA  in  the 
1980s.  He  shows  how  the  modern  para¬ 
digm  is  to  combine  Technology,  Integration,  and  Manufacturing 
Readiness  Levels  into  a  single  metric— System  Readiness  Level— 
which  can  be  used  as  a  more  robust  indicator  of  the  maturity  of  the 
technology  transfer  process. 

“Tailoring  a  Large  Organization’s  Systems  Engineering  Process 
to  Meet  Project-Specific  Needs”  by  Matthew  Graviss,  Shahram 
Sarkani,  and  Thomas  A.  Mazzuchi  shows  how  a  customized, 
streamlined  approach  to  systems  engineering  can  be  performed 
using  a  rule-based  process  that  allows  project  flexibility  while  also 
adhering  to  an  organization’s  top-level  policies. 

Patrick  Clowney,  Jason  Dever,  and  Steven  Stuban,  in  “Department  of 
Defense  Acquisition  Program  Terminations:  Analysis  of  11  Program 
Management  Factors,”  discuss  the  results  of  their  surveys  of  acqui¬ 
sition  program  managers  in  the  DoD,  industry,  and  consultancies. 
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which  evaluated  the  factors  that  have  the  greatest  influence  on  DoD 
program  termination,  and  how  the  viewpoints  of  these  three  groups 
differed  or  aligned. 

Finally,  “Wartime  Construction  Project  Outcomes  as  a  Function  of 
Contract  Type”  by  Ryan  Hoff,  Gregory  Hammond,  Peter  Feng,  and 
Edward  White  shows  that  in  wartime  contracting,  although  cost- 
plus-fixed-fee  contracts  exhibited  greater  cost  and  schedule  growth 
than  firm-fixed-price  contracts,  neither  the  finished  quality  nor  the 
associated  construction  risks  differed  greatly  between  them. 

The  featured  book  in  this  issue’s  Defense  Acquisition  Professional 
Reading  List  is  Predator:  The  Secret  Origins  of  the  Drone  Revolution 
by  Richard  Whittle,  reviewed  by  Julien  Demotes-Mainard  at 
Avascent  Europe. 
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agenda  annually,  using  inputs  from  subject  matter  experts  across 
those  sectors.  Topics  are  periodically  vetted  and  updated  by  the 
DAU  Center’s  Research  Advisory  Board  to  ensure  they  address 
current  areas  of  strategic  interest. 

The  purpose  of  conducting  research  in  these  areas  is  to  provide 
solid,  empirically  based  findings  to  create  a  broad  body  of  knowl¬ 
edge  that  can  inform  the  development  of  policies,  procedures,  and 
processes  in  defense  acquisition,  and  to  help  shape  the  thought  lead¬ 
ership  for  the  acquisition  community.  Most  of  these  research  topics 
were  selected  to  support  the  DoD’s  Better  Buying  Power  Initiative 
(see  http://bbp.dau.mil).  Some  questions  may  cross  topics  and  thus 
appear  in  multiple  research  areas. 

Potential  researchers  are  encouraged  to  contact  the  DAU  Director 
of  Research  (research@dau.mil)  to  suggest  additional  research 
questions  and  topics.  They  are  also  encouraged  to  contact  the 
listed  Points  of  Contact  (POC),  who  maybe  able  to  provide  general 
guidance  as  to  current  areas  of  interest,  potential  sources  of  infor¬ 
mation,  etc. 
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Competition  POCs 

•  John  Cannaday,  DAU:  john.cannaday@dau.mil 

•  Salvatore  Cianci,  DAU:  salvatore.cianci@dau.mil 

•  Frank  Kenlon  (global  market  outreach),  DAU:  frank. 
kenlon@dau.mil 

Measuring  the  Effects  of  Competition 

•  What  means  are  there  (or  can  be  developed)  to  measure 
the  effect  on  defense  acquisition  costs  of  maintaining 
the  defense  industrial  base  in  various  sectors? 

•  What  means  are  there  (or  can  be  developed)  of  mea¬ 
suring  the  effect  of  utilizing  defense  industrial 
infrastructure  for  commercial  manufacture,  and  in 
particular,  in  growth  industries?  In  other  words,  can 
we  measure  the  effect  of  using  defense  manufacturing 
to  expand  the  buyer  base? 

•  What  means  are  there  (or  can  be  developed)  to  deter¬ 
mine  the  degree  of  openness  that  exists  in  competitive 
awards? 

•  What  are  the  different  effects  of  the  two  best  value 
source  selection  processes  (trade-off  vs.  lowest  price 
technically  acceptable)  on  program  cost,  schedule,  and 
performance? 

Strategic  Competition 

•  Is  there  evidence  that  competition  between  system 
portfolios  is  an  effective  means  of  controlling  price 
and  costs? 

•  Does  lack  of  competition  automatically  mean  higher 
prices?  For  example,  is  there  evidence  that  sole  source 
can  result  in  lower  overall  administrative  costs  at  both 
the  government  and  industry  levels,  to  the  effect  of 
lowering  total  costs? 

•  What  are  the  long-term  historical  trends  for  compe¬ 
tition  guidance  and  practice  in  defense  acquisition 
policies  and  practices? 
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•  To  what  extent  are  contracts  being  awarded  non- 
competitively  by  congressional  mandate  for  policy 
interest  reasons?  What  is  the  effect  on  contract  price 
and  performance? 

•  What  means  are  there  (or  can  be  developed)  to  deter¬ 
mine  the  degree  to  which  competitive  program  costs 
are  negatively  affected  by  laws  and  regulations  such  as 
the  Berry  Amendment,  Buy  America  Act,  etc.? 

•  The  DoD  should  have  enormous  buying  power  and  the 
ability  to  influence  supplier  prices.  Is  this  the  case? 
Examine  the  potential  change  in  cost  performance 
due  to  greater  centralization  of  buying  organizations 
or  strategies. 

Effects  of  Industrial  Base 

•  What  are  the  effects  on  program  cost,  schedule,  and 
performance  of  having  more  or  fewer  competitors? 
What  measures  are  there  to  determine  these  effects? 

•  What  means  are  there  (or  can  be  developed)  to  measure 
the  breadth  and  depth  of  the  industrial  base  in  various 
sectors  that  go  beyond  simple  head-count  of  providers? 

•  Has  change  in  the  defense  industrial  base  resulted  in 
actual  change  in  output?  How  is  that  measured? 

Competitive  Contracting 

•  Commercial  industry  often  cultivates  long-term,  exclu¬ 
sive  (noncompetitive)  supply  chain  relationships.  Does 
this  model  have  any  application  to  defense  acquisition? 
Under  what  conditions/circumstances? 

•  What  is  the  effect  on  program  cost,  schedule,  and 
performance  of  awards  based  on  varying  levels  of 
competition:  (a)  “Effective”  competition  (two  or  more 
offers);  (b)  “Ineffective”  competition  (only  one  offer 
received  in  response  to  competitive  solicitation);  (c) 
split  awards  vs.  winner  take  all;  and  (d)  sole  source. 


XVI 


Improve  DoD  Outreach  for  Technology  and  Products 
from  Global  Markets 

•  How  have  militaries  in  the  past  benefitted  from  global 
technology  development? 

•  How/why  have  militaries  missed  the  largest  techno¬ 
logical  advances? 

•  What  are  the  key  areas  that  require  the  DoD’s  focus  and 
attention  in  the  coming  years  to  maintain  or  enhance 
the  technological  advantage  of  its  weapon  systems  and 
equipment? 

•  What  types  of  efforts  should  the  DoD  consider  pursu¬ 
ing  to  increase  the  breadth  and  depth  of  technology 
push  efforts  in  DoD  acquisition  programs? 

•  How  effectively  are  the  DoD’s  global  science  and  tech¬ 
nology  investments  transitioned  into  DoD  acquisition 
programs? 

•  Are  the  DoD’s  applied  research  and  development  (i.e., 
acquisition  program)  investments  effectively  pursuing 
and  using  sources  of  global  technology  to  affordably 
meet  current  and  future  DoD  acquisition  program 
requirements?  If  not,  what  steps  could  the  DoD  take 
to  improve  its  performance  in  these  two  areas? 

•  What  are  the  strengths  and  weaknesses  of  the  DoD’s 
global  defense  technology  investment  approach  as 
compared  to  the  approaches  used  by  other  nations? 

•  What  are  the  strengths  and  weaknesses  of  the  DoD’s 
global  defense  technology  investment  approach  as 
compared  to  the  approaches  used  by  the  private  sec¬ 
tor— both  domestic  and  foreign  entities  (companies, 
universities,  private -public  partnerships,  think  tanks, 
etc.)? 

•  How  does  the  DoD  currently  assess  the  relative  benefits 
and  risks  associated  with  global  versus  U.S.  sourcing 
of  key  technologies  used  in  DoD  acquisition  programs? 
How  could  the  DoD  improve  its  policies  and  procedures 
in  this  area  to  enhance  the  benefits  of  global  technology 
sourcing  while  minimizing  potential  risks? 
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•  How  could  current  DoD/U.S.  Technology  Security  and 
Foreign  Disclosure  (TSFD)  decision-making  policies 
and  processes  be  improved  to  help  the  DoD  better  bal¬ 
ance  the  benefits  and  risks  associated  with  potential 
global  sourcing  of  key  technologies  used  in  current  and 
future  DoD  acquisition  programs? 

•  How  do  DoD  primes  and  key  subcontractors  currently 
assess  the  relative  benefits  and  risks  associated  with 
global  versus  U.S.  sourcing  of  key  technologies  used  in 
DoD  acquisition  programs?  How  could  they  improve 
their  contractor  policies  and  procedures  in  this  area 
to  enhance  the  benefits  of  global  technology  sourcing 
while  minimizing  potential  risks? 

•  How  could  current  U.S.  Export  Control  System  deci¬ 
sion-making  policies  and  processes  be  improved  to 
help  the  DoD  better  balance  the  benefits  and  risks 
associated  with  potential  global  sourcing  of  key  tech¬ 
nologies  used  in  current  and  future  DoD  acquisition 
programs? 

Comparative  Studies 

•  Compare  the  industrial  policies  of  military  acquisition 
in  different  nations  and  the  policy  impacts  on  acquisi¬ 
tion  outcomes. 

•  Compare  the  cost  and  contract  performance  of  highly 
regulated  public  utilities  with  nonregulated  “natural 
monopolies  ,”  e.g.,  military  satellites,  warship  building, 
etc. 

•  Compare  contracting/competition  practices  between 
the  DoD  and  complex,  custom-built  commercial  prod¬ 
ucts  (e.g.,  offshore  oil  platforms). 

•  Compare  program  cost  performance  in  various  market 
sectors:  highly  competitive  (multiple  offerors),  limited 
(two  or  three  offerors),  monopoly? 

•  Compare  the  cost  and  contract  performance  of  mil¬ 
itary  acquisition  programs  in  nations  having  single 
“purple”  acquisition  organizations  with  those  having 
Service-level  acquisition  agencies. 
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Application  of  System  and  Integration 

READINESS  LEVELS 

to  Department  of  Defense 

RESEARCH  and  DEVELOPMENT 


i 


Sean  Ross 


Technology  Readiness  Level  only  tells  part  of  the  story  of 
system  maturation.  As  component  technologies  are  devel¬ 
oped  to  become  part  of  systems,  there  are  also  integration 
and  manufacturing  issues  to  consider.  This  article  improves 
upon  the  System  and  Integration  Readiness  Level  concepts 
previously  developed  by  B.  J.  Sauser  et  al.,  combines  the 
concepts  of  Technology,  Integration,  and  Manufac 
turing  Readiness  Levels,  adapted  for  use  in  defense 
acquisition,  into  a  single  metric— System  Readiness 
Level.  This  metric  can  then  be  used  as  an  indicator 
to  identify  areas  for  resource  allocation  to  enable 
the  most  efficient  path  to  technology  transition 
and  to  prevent  premature  system  advancement. 


Keywords:  Technology  Readiness  Level  (TRL),  Integration 
Readiness  Level  (IRL),  Manufacturing  Readiness  Level  (MRL), 
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In  an  ideal  world,  a  component  technology  would  develop  concurrently 
with  its  interfaces  and  its  ability  to  be  manufactured.  In  the  real  world, 
technologies  lead  both  their  interfaces  and  manufacturing  infrastructure. 
For  example,  motorcycles  were  first  made  with  fixed  foot-pegs  until  some 
rather  spectacular,  spin-out  wrecks  occurred,  prompting  folding  foot -pegs. 
The  human- motorcycle  interface  maturity  followed  the  technical  maturity 
at  the  expense  of  safety.  Early  airplanes  were  made,  one-at-a-time,  with 
bicycle  manufacturing  equipment.  The  manufacturing  maturity  lagged  the 
technology.  The  competing  pitfall  in  system  development  is  the  premature 
advancement  of  a  technology  to  the  next  level  of  development  in  advance  of 
its  interfaces,  such  as  the  current  state  of  the  F-35  program.  Although  the 
program  is  in  late  stage  development,  interface  and  component  technology 
issues  are  still  emerging  that  are  preventing  full  operational  capability 
(Bender,  2015).  We  can  do  a  better  job  by  minimizing  the  gap  between 
interface,  manufacturing,  and  technology  maturity.  Integration  and  sys¬ 
tem  readiness  are  not  yet  implemented  in  any  formal  way  Department  of 
Defense  (DoD)-wide. 

This  article  explains  a  method  to  combine  Technology  Readiness  Level 
(TRL)  (See  Appendix,  Table  A-l),  Integration  Readiness  Level  (IRL),  and 
Manufacturing  Readiness  Level  (MRL)  (See  Appendix,  Table  A-2)  into  a 
single  metric— System  Readiness  Level  (SRL)— that  can  provide  guidance  to 
decision  makers  during  the  technology  maturation  process.  Such  guidance 
can  minimize  the  delays  and  mishaps  likely  to  occur  when  interfaces  and 
manufacturing  significantly  lag  their  component  technologies. 


Background 

The  DoD  Research,  Development,  Test  and  Evaluation  budget  is  sub¬ 
divided  into  seven  separate  activities:  basic  research;  applied  research; 
advanced  technology  development;  advanced  component  development  and 
prototypes;  system  development  and  demonstration;  research,  develop¬ 
ment,  test  and  evaluation  (RDT&E)  management  support;  and  operational 
systems  development,  i.e.,  the  DoD  categories  of  funding  and  technology 
development  (Appendix,  Table  A- 3).  These  seven  activities  are  designated 
as  DoD  6.1  through  6.7.  This  article  incorporates  the  6.1  through  6.7  levels 
of  funding  and  appropriate  levels  of  maturity  so  that  the  same  metric  can  be 
used  throughout  the  acquisition  life  cycle.  Verbal  definitions  of  TRL,  MRL, 
IRL,  and  SRL  are  included  at  the  end  of  the  article. 
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Sauser,  Ramirez-Marquez,  and  Devanandham,  and  Dimarzio  (2007),  and 
Sauser,  Ramirez-Marquez,  Magnaye,  and  Tan  (2008a)  furthered  the  con¬ 
cepts  of  TRL  to  include  IRL  and  SRL  (Sauser,  Forbes,  Long,  &  McGrory, 
2009;  Sauser,  Gove,  Forbes,  &  Ramirez-Marquez,  2010).  These  approaches 
emphasize  that  the  interfaces  between  subsystems  are  every  bit  as  import¬ 
ant  as  the  subsystems  themselves,  and  that  no  system  can  be  deemed  ready 
for  deployment  based  on  the  component  technologies  alone. 


Method 

Sauser’s  basic  approach  is  to  imagine  a  system  composed  of  component 
technologies  from  1  to  n,  each  with  a  TRL  as  shown  in  equation  (1)  and 
Figure  1. 


TRL  =  trl.=  {trl  trL  ■■■  trl  } 

i  v  1  2  nr 


(1) 


Mathematical  Note.  A  list  of  symbols  or  numbers  in  braces 
represents  a  vector.  A  subscripted  symbol  indicates  one 
element  out  of  a  vector.  A  number  without  subscripts  indi¬ 
cates  the  whole  vector  quantity.  Lower  case  is  used  for 
normalized  quantities. 


FIGURE  1.  A  SYSTEM  AS  A  COLLECTION  OF  COMPONENT 
TECHNOLOGIES 


Note.  (Sauser,  2008) 
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For  example,  a  motorcycle  can  be  viewed  as  an  engine,  power  train,  exhaust, 
electrical  system,  cooling  system,  saddle,  suspension,  wheels,  gauges,  steer¬ 
ing,  headlamp,  etc. 

Each  component  technology  has  a  potential  interface  with  each  other  com¬ 
ponent  and  with  the  external  environment,  including  the  possibility  of  an 
interface  going  both  ways,  as  shown  in  equation  (2).  For  simplicity,  Figure  2 
shows  the  interfaces  with  double  arrows,  as  if  irl10  =  irlzl,  which  need  not  be 
the  case.  IRL  must  be  expressed  as  a  two-dimensional  matrix  rather  than 
a  one-dimensional  vector.  The  vector  is  generally  square— with  the  same 
number  of  rows  and  columns.  The  diagonal  of  the  matrix  is  not  used  since 
a  technology  always  works  with  itself. 


1 

f  X 

irl 

12 

irl13  - 

■  irl 

IRL  =  irl..=  J 

V  1 

irl21 

X 

irl2 1  - 

■  irl. 

1 

L  irL 

irl  , 

nl 

irL  ■■ 

■  irl 

1 

' 

» 


/ 


(2) 


FIGURE  2.  SYSTEM  AS  A  COLLECTION  OF  INTERFACES  AND 
COMPONENT  TECHNOLOGIES 


In  the  Sauser  approach.  The  IRL  matrix  and  the  TRL  vector  are  multiplied 
together  as  a  vector  product  (U.S.  Navy,  2009,  p.  35)  to  form  an  SRL  vector 
that  can  be  averaged  for  an  overall  SRL  (Sauser,  Verma,  Ramirez-Marequez, 
Gove,  2006,  p.  A-12;  Sauser  et  al„  2007,  p.  681;  U.S.  Navy,  2009,  p.  33).  Note 
that  this  paper  shows  matrix  notation  in  both  reduced  tensor  notation  and 
matrix  notation  as  a  convenience  for  a  multidisciplinary  audience.  SRL , 

- >  _  J 

[SRL]  and  SRL  all  refer  to  the  same  vector  entity  and  all  versions  of  equation 
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(3)  show  the  same  tensor/matrix  operation  in  different  notation.  Equation 

(4)  shows  the  Sauser  formula  for  SRL.  Computational  and  practical  exam¬ 
ples  of  all  formulas  will  be  shown  in  the  examples  section. 


SRL.  =  IRL..  TRV  =  IRL1  TRL 1  +  IRL..  TRL2  +  +  =  IRL  .  TRLn 


(3a) 


[SRL] 


"  SRL1  " 
SRLZ 

-  SRL3  . 

'  IRL, ,  TRL,  +  IRL, „  TRL„  +  -  +  IRL,  TRL 

11  1  12  2  In  n 

IRL,  ,  TRL+IRL...TRL+-  +  IRL „  TRL 


IRL  ,  TRL,  +  IRL  0TRL+  -  +  IRL  TRL  „ 

1  nl  1  n2  2  nn  n 1 


(3b) 


(  irlu  irllz  irl13  -  irlln 


SRL  =  < 


irl  irl  irl 

12  22  21 


13  21 


irl 


'  tr\  ' 

tH2  * 

/ 

-  trl  * 

(3c) 


SRL=j-  Z  srl 

■LV  j=ltoN 


(4) 


As  shown  in  equations  (3a)  and  (3b),  the  Sauser  mathematics  views  a  com¬ 
ponent  of  SRL  (SRL  . )  as  being  based  upon  a  single  interface  type  and  its 
associated  technologies;  the  SRL1  component  includes  TRL  TRLZ,  etc., 
and  all  of  the  IRLln  rather  than  a  technology-centric  approach 
that  included  TRL1  with  all  its  interfaces.  The  inter- 
face-centric  approach  is  graphically  shown  in  Figure  3 
and  contrasted  with  a  technology-centric  approach  in 
Figure  4  using  a  motorcycle.  The  mechanical  compo¬ 
nent  of  SRL  (SRL  ,  .  ,)  in  the  Sauser 
approach  for  a  motorcycle  would  be 
based  upon  the  mechanical-en 
gine,  mechanical-headlamps, 
mechanical-saddle,  mechan¬ 
ical-tires,  etc.,  interfaces.  The 
interface-centric  approach  has 
some  serious  limitations  as  will 
be  covered  in  the  next  sections. 
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FIGURE  3.  A  COMPONENT  OF  SRL  BASED  UPON  A  SINGLE  INTERFACE 
AND  ITS  ASSOCIATED  TECHNOLOGIES 


Note.  (Sauser  et  al.,  2007) 


FIGURE  4.  LEFT:  EXAMPLE  FROM  A  MOTORCYCLE:  INTERFACE¬ 
CENTRIC  APPROACH;  RIGHT:  TECHNOLOGY-CENTRIC  APPROACH 
PRESENTED  IN  THIS  ARTICLE 


Note.  Left:  (Sauser,  2008).  Right:  Ross,  S.  (2016).  Application  of  System  and  Integration 
Readiness  Levels  to  Department  of  Defense  Research  and  Development.  Defense 
Acquisition  Research  Journal,  23( 3),  In  Print. 


The  average  of  the  SRL  vector,  equation  (4),  describes  how  mature  the  sys¬ 
tem  is.  The  Sauser  approach  may  make  sense  for  a  single  mission  or  project, 
such  as  the  deployment  of  a  new  software  system.  However,  it  has  some 
serious  drawbacks  for  use  in  research  and  development  where  planners 
need  to  decide  what  technologies  to  develop  for  the  eventual  deployment  of 
a  new  platform,  weapon,  or  system.  First,  SRL,  as  defined  in  the  U.S.  Navy’s 
Littoral  Combat  Ship  Mission  Module  Program  System  Maturity  Assessment 
Guide  (2009),  is  interface- centric  as  opposed  to  component- centric.  The 
Sauser  definition  shows  each  interface  with  its  associated  technologies 
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rather  than  each  technology  with  its  associated  interfaces.  Second,  SRL  as 
defined  by  Sauser,  has  no  clear  meaning  assigned  to  a  given  numerical  value. 
In  one  presentation  (Sauser,  Ramirez-Marquez,  Magnaye,  &  Tan,  2008b), 
SRL  is  defined  along  a  value  from  0  to  1  with  five  unequal  intermediate  levels 
and  no  verbal  definitions  akin  to  those  for  TRL,  IRL,  and  MRL.  This  gives 
SRL  a  different  kind  of  scale  than  IRL  and  TRL,  which  are  clearly  defined 
such  that  1  is  a  concept  and  9  is  full  deployment.  Third,  the  Sauser-defined 
SRL  only  has  meaning  at  the  full  system  level.  The  interface-centric  compo¬ 
nents  of  the  SRL  vector  give  no  guidance  to  component  developers.  Finally, 
the  definitions  of  IRL  tend  to  be  information  technology  (IT)-centric, 
emphasizing  control  and  information.  IRL  needs  to  be  applicable  to  a  wide 
variety  of  interfaces,  including  mechanical,  thermal,  electrical,  structural, 
and  control  interfaces  as  well  as  logistics,  policy,  and  other  ‘-ility’  and  mis¬ 
sion  interfaces. 

Characteristics  of  a  Useful  System  Readiness  Level  Metric 

A  useful  metric  will  be  defined  so  as  to  give  a  clear  indication  for  plan¬ 
ning  resource  allocation.  SRL  and  IRL,  as  metrics,  can  be  useful  if  they  are 
defined  correctly.  The  author  proposes  the  following  criteria  for  a  useful 
SRL  and  IRL  metric. 

1.  IRL  definitions  should  be  applicable  to  a  wide  variety  of 
technologies. 

2.  SRL  should  be  defined  such  that  SRL=1  is  a  concept  and  SRL=9 
is  a  mature,  deployed  system  on  the  same  basic  scale  as  TRL, 
MRL,  and  IRL. 

3.  SRL  should  equal  TRL  when  the  interfaces  are  developed  con¬ 
currently  with  the  components,  and  should  be  less  than  TRL 
when  interfaces  are  less  mature  than  the  components.  This 
will  give  planners  a  clear  metric  that  lets  them  know  that  it  is 
time  to  transition  funding  into  more  interface-centric  devel¬ 
opment  or  to  proceed  with  component  technology  maturation. 

4.  SRL  should  be  technology-  or  component- centric,  not 
interface- centric.  This  makes  it  clear  when  a  particular 
subcomponent  is  not  able  to  progress  further  toward  imple¬ 
mentation  due  to  an  interface  or  manufacturing  issue. 

5.  SRL  should  include  MRL,  TRL,  and  IRL. 
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6.  SRL  should  be  applicable  to  a  wide  variety  of  technical  matur¬ 
ities  (see  Appendix,  Table  A- 3),  including  basic  research  (6.1 
funding);  applied  research  (6.2  funding);  advanced  technology 
development  (6.3  funding);  advanced  component  develop¬ 
ment  and  prototypes  (6.4  funding);  system  development  and 
demonstration  (6.5  funding);  and  operational  systems  devel¬ 
opment  (6.7  funding),  i.e.,  the  DoD  categories  of  funding  and 
technology  development.  Note  that  6.6  funding  is  not  included 
because  it  is  for  management  activities  and  not  tied  to  a  level 
of  technical  maturity. 

7.  SRL  must  be  defined  in  such  a  way  as  to  avoid  maturity  in  one 
component  overshadowing  immaturity  in  another  (Kujawski, 
2010)  and  giving  the  illusion  that  the  system  is  ready  to  prog¬ 
ress.  This  implies  that  SRL  should  never  be  able  to  be  greater 
than  TRL  at  either  the  system  or  component  level. 


Proposed  System  Readiness  Level  Metric 

The  author  proposes  that  a  more  useful  way  to  arrange  MRL,  TRL,  and 
IRL  is  as  a  series  of  normalized  dot  products,  rather  than  vector  products 
(Sauser  et  al.,  2008a,  p.  47).  This  changes  the  view  of  the  components  of 
SRL  from  being  interface-centric  to  being  technology-centric,  as  shown 
in  the  contrast  between  Figure  3  and  Figure  5,  and  between  the  right  and 
left  sides  of  Figure  4.  The  SRL  components  are  equal  to  the  product  of  the 
normalized  MRL,  the  TRL,  and  the  mean  of  the  normalized  IRL, 
as  shown  in  Table  1.  In  the  notation  that  follows,  upper  case  is 
reserved  for  standard  (i.e.,  verbal)  definitions  and  lower  case 
is  for  normalized  quantities.  Note  that  the  word  ‘system’ 
in  this  article  refers  to  a  generic  system— anything  that 
can  be  usefully  viewed  as  being  composed  of  parts,  rather 
than  specifically  as  a  deployed  military  asset.  Likewise, 
the  term  ‘component’  refers  to  the  parts  that  make  up 
a  larger  grouping  rather  than  exclusively  as  a  line-re¬ 
placeable  item  with  a  specific  part  number.  The  term 
interface  should  be  viewed  in  the  broad  sense  of  the 
word  to  also  include  the  external  environment— the 
‘ilities’  (availability,  maintainability,  vulnerability, 
reliability,  supportability,  etc.)  and  the  DOTmLPF-P 
(Doctrine,  Organization,  Training,  materiel,  Leadership 
and  Education,  Personnel,  Facilities-Policy). 
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FIGURE  5.  A  COMPONENT  OF  SRL  BASED  UPON  A  SINGLE 
TECHNOLOGY  AND  ITS  INTERFACES 


1 _ 1 

Note.  Ross,  S.,  (2016).  Application  of  System  and  Integration  Readiness  Levels  to 
Department  of  Defense  Research  and  Development.  Defense  Acquisition  Research 

Journal,  23(3),  In  Print. 

TABLE  1.  NORMALIZED  INTEGRATION  READINESS 

LEVEL  DEFINITIONS 

IRL  = 

Integration  readiness  level  scalar 

IRL  = 

Jk 

IRL  for  the  interface  between  technology  j  and  technology  k 

irL  = 

Jk 

normalized  IRL  for  interface  between  technology  j  and 
technology  k 

irL  = 

jj 

1,  the  interface  always  works  with  itself 

irl  = 

Jk 

irlkj,  the  interface  works  both  ways.  It  may  be  useful  for  some 
systems  to  break  the  IRL  apart  into  two  components.  For 
purposes  of  this  article,  the  author  assumes  that  if  the  motor- 
fuel  interface  works,  so  does  the  fuel-motor  interface. 

irl  = 

IRL/i* 

Research 

level 

6.1  6.2  6.3  6.4  6.5  6.6  6.7 

j* 

1  2  3  5  6  7  9 

n 

iri  =  mean\_iri. ]  —  ^  /'/•/.. 

j  =  1 

To  have  SRL  equal  to  TRL  when  IRL  and  MRL  are  at  commensurate  levels 
of  development  requires  normalized  versions  of  IRL  and  MRL  scaled  to 
the  level  of  research.  Basic  research  (6.1  funding)  should  have  a  goal  of  an 
IRL  of  1  (Interface  identification)  and  MRL  of  2  (Manufacturing  concepts 
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identified),  so  that  the  normalized  mrl  and  irl  equal  1  when  the  appropriate 
levels  of  IRL  and  MRL  are  reached.  Likewise,  system  development  and 
demonstration  (6.5  funding)  should  have  as  its  goal  an  IRL  of  6  (interface 
control)  and  an  MRL  of  6  (prototype  in  a  production-relevant  environ¬ 
ment)  so  the  normalized  mrl  and  irl  equal  1  when  the  appropriate  levels 
are  reached.  The  signal  to  proceed  to  the  next  step  in  system  development 
occurs  when  SRL  equals  TRL,  indicating  that  the  interfaces  and  manufac¬ 
turing  base  are  at  a  commensurate  level  of  development  with  the  component 
technologies.  The  nomenclature  and  definitions  for  normalized  IRL  are 
shown  in  Table  1.  The  normalization  factors  are  chosen  to  be  consistent  with 
the  funding  categories  listed  in  the  DoD  Research,  Development,  Test  and 
Evaluation  (RDT&E)  budget  (Appendix,  Table  A- 3).  Different  communities 
may  have  differing  levels  of  MRL,  TRL,  and  IRL  goals  vs.  acquisition  stage 
so  that  the  normalization  factors  are  intended  as  starting  suggestions.  It 
would  also  be  viable  to  have  normalization  factors  based  on  the  DoD  5000.02 
Model  1  (DoD,  2015). 

The  normalized  IRLs  associated  with  a  particular  technology  need  to  be 
averaged  to  come  up  with  a  representative  number  indicating  how  well 
that  particular  technology  relates  to  the  other  subsystems  or  technologies 
in  the  system.  The  irl.  accomplish  this.  Note  that  the  normalization  factors 
‘reset’  the  metric  at  each  level  of  maturity,  which  reduces  the  possibility  of 
one  very  mature  component  masking  a  less  mature  one  in  the  metric.  MRL 
normalizations  and  definitions  are  shown  in  Table  2.  Note  that  the  normal¬ 
ized  MRL  (mrl)  does  not  replace  the  existing  MRL,  but  is  an  intermediate 
step  needed  for  SRL  calculation  as  is  the  normalized  IRL  (irl). 


TABLE  2.  NORMALIZED  MANUFACTURING  READINESS 

LEVEL  DEFINITIONS 

MRL  = 

j 

MRL  for  technology  j 

mrl.  = 

J 

normalized  MRL  for  technology  j 

mrl.  = 

J 

MRL/m* 

Research 

level 

6.1  6.2  6.3  6.4  6.5  6.6 

6.7 

m* 

2  3  4  5  6  8 

10 

The  SRL  metric  is  formed  by  multiplying  the  normalized  MRL,  the  TRL, 
and  the  mean  of  the  normalized  IRL  in  a  scalar  contraction  (dot  product) 
such  that  each  component  SRL.  has  a  value  from  1  to  TRL  as  does  the  scalar 
SRL.  System  readiness  definitions  and  nomenclature  are  shown  in  Table  3. 
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TABLE  3.  SYSTEM  READINESS  LEVEL  DEFINITIONS 

AND  NOMENCLATURE 

SRL  = 

System  readiness  level  scalar,  the  mean  of  the  system  readiness 
levels  for  all  component  technologies 

SRL.  = 
j 

System  readiness  level  for  component  j 

SRL  -  mrl.TRL.lrl. 

1  III 

SRL  =  mean\_{mr!^  mrl2 ...  mrln}.{TRL ,  TRL2 ...  TRLn}.{irl \  irl2 ...  irln}\ 


Numerical  Examples 

For  simplicity  and  clarity,  this  article  shows  three  numerical-only 
examples  using  a  hypothetical  system  with  three  technologies,  as  shown 
in  Figure  6. 


FIGURE  6.  EXAMPLE  SYSTEM  WITH  THREE 
COMPONENT  TECHNOLOGIES 


Each  technology  has  an  associated  TRL  and  MRL.  Each  Interface  has  an 
associated  IRL.  Notationally,  this  will  be  of  the  form  shown  in  equations 
(6),  (7),  and  (8). 


TRL.  =  {TRLi;  TRL2,  TRL3} 


(6) 


MRL.  =  {MRLi;  MRL2,  MRL3}. 


(7) 
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l  x 

X 

(8) 


Early  Technology  with  Adequate  Interfaces 

The  author  assumes  a  simple,  three-component  system,  with  com¬ 
ponents  1,  2,  and  3  funded  at  the  6.2— applied  research  level— using  the 
following  values  for  MRL,  TRL,  and  IRL  shown  in  equations  (9),  (10),  (11), 
and  (12). 


MRL  ={2,3,3} 


TRL  ={3,2,4} 


TRL  =  mean[3,2,4]  =  3 


(9) 

(10) 

(ID 


X  irL12  IRL13  =2 
IRL,=  x  X  IRL23  =2 
XX  X 


(12) 


The  first  step  is  to  calculate  the  normalized  mrl  and  irl  using  the  equations 
from  Tables  1  and  2.  Because  this  is  6.2  funded,  the  m*  normalization  factor 
is  3  from  Table  2,  indicating  that  we  expect  6.2  funded  technologies  to  be  at 
an  MRL  of  3  before  progressing.  Likewise,  the  i*  normalization  factor  is  2 
from  Table  1.  Normalized  values  are  shown  in  equations  (13),  (14),  and  (15). 


mrl=MRL./mM2,3,3}/3={0.66,l,l} 


f  X 

0.5 

1  \ 

irl..  =  IRL../2  =  ^  X 

1J  1J '  J 

X 

1  l 

l  x 

X 

X  , 

(13) 


(14) 
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ir Z1  =  mean[irl12,  irl13]  =  0.75 ,  irl2  =  mean[irl12,  irl23]  =  0.75,  irl3  = 
mean[irl23,  irl13]  =  1. 


(15) 


The  SRL  vector  is  calculated  from  the  products  of  the  normalized  mrl, 
average  normalized  irl,  and  TRL  vectors  using  the  formulas  from  Table  3, 
shown  in  equations  (16)  and  (17). 

SRL.=  mrl.  TRL  irl.  =  {0.66*3*0.75,1*2*0.75,1*4*1}  =  {1.49,1.5,4} 

(16) 


SRL  =  mean[SRL.]  =  2.33 


(17) 


Analysis 

SRL  =  2.33  while  the  average  TRL  is  3,  indicating  a  slight  lag  in  at  least 
one  interface.  From  the  normalized  MRL,  one  can  conclude  that  the 
system  is  at  a  mostly  appropriate  level  of  manufacturing 
readiness  with  two  components  at  an  mrl  of  1  and 
one  at  0.66.  SRL1  and  SRL;i  are  at  1.5,  slightly  lag¬ 
ging  behind  the  technology  readiness  of  2  and  3 
due  to  some  interface  development  that  needs  to 
occur.  SRL3  =  TRL3  =  4  indicates  that  this  tech¬ 
nology  is  at  an  appropriate  level  of  interface 
and  manufacturing  readiness.  The  metric  indi¬ 
cates  to  management  that  it  is  time  to  devote 
additional  resources  to  the  interfaces  of 
technologies  1  and  2  before  pushing  ahead  in 
further  component  or  system  development. 

It  is  very  important  to  conduct  the  early 
phases  of  interface  readiness,  which 
involve  subject  matter  experts  from  dif¬ 
ferent  fields  exchanging  information  and 
ensuring  that  there  exists  an  interface 
solution.  If  this  is  skipped,  then  at  the 
demonstration  and  prototyping  levels 
of  6.4  research,  many  technology  choices 
must  be  revisited  because  the  technologies  have 
matured  separately  and  are  becoming  incompatible.  Revisiting 
technology  choices  may  then  result  in  program  delays,  cost  overruns,  or 
mad  scrambles  to  prepare  for  demonstrations  or  program  cancellations. 
The  classic  case  of  this  is  thermal  management,  when  a  new  technology 
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becomes  available  with  thermal  management  as  an  afterthought,  and  the 
legacy  platform  for  which  it  is  intended  becomes  overwhelmed  with  the 
new  thermal  load.  The  thermal  issues  associated  with  5th  generation  air¬ 
craft  (Majumdar  &  Kjelgaard,  2015)  are  a  result  of  thermal  interface  as  an 
afterthought.  The  opposite  appears  to  be  happening  in  the  semiconductor 
industry  in  which  thermal  management  is  a  very  active  area  of  research  in 
anticipation  of  higher  thermal  loads  on  microchips  in  the  near  future. 

Mid-level  Technology  with  Lagging  Interfaces 

In  a  more  abbreviated  form  than  the  previous  example,  we  assume  a 
simple,  three-component  system  with  components  1,  2,  3  funded  at  the 
6.4  -  demonstration  level.  MRL .  =  {3,5,5};  TRL.  =  {6,4,5};  TRL  =  5,  and  the 
IRL  has  the  following  values:  IRL12  =  3,  IRL13  =  2,  and  IRL23  =  2.  The  SRL 
calculation  is  as  follows: 

mrl.  =  {0.6, 1,1}.  irllz  =  3/5,  irl13  =  2/5,  and  irl23  =  2/5,  irlt  =  mean[irll2,  irll3]  = 
0.5 ,  irl2  =  mean [irl12,  irZ,3]  =  0.5,  irl3  =  mean[irZ23,  irl  ^  =  0.4. 

SRL.=  mrl.  TRL. irl.  =  {0.6*6*0.5,1M*0.5,1*5*0.4}  =  {1.8, 2, 2} 

SRL  =  mean[SRL.]  =  1.3 


It  makes  no  sense  to  continue  and  pursue  more  ma¬ 
ture  technology  that  may  or  may  not  work  in  the 
intended  environment  or  with  the  other  subsystems. 


Analysis 

The  fact  that  SRL  =  1.3,  but  there  are  TRLs  at  6  and  4  and  an  average 
TRL  of  5,  alerts  management  there  are  serious  manufacturing  and  interface 
issues,  probably  due  to  neglect  in  early  technical  development.  Note  that  the 

is  0.6  and  is  slightly  lower  than  the  other  two;  the  SRL.  are  very  nearly 
all  at  2;  and  the  TRL.  are  quite  high— at  6, 4,  and  5— due  to  the  irl  being  much 
lower.  This  alerts  management  that  emphasis  needs  to  be  placed  on  develop¬ 
ing  interfaces.  Further  component  maturation  is  very  risky  and  very  likely 
counter-productive.  It  makes  no  sense  to  continue  and  pursue  more  mature 
technology  that  may  or  may  not  work  in  the  intended  environment  or  with 
the  other  subsystems.  This  system  is  headed  toward  program-killing  safety, 
thermal,  control,  electrical,  or  other  integration  and  deployment  issues. 
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Advanced  Technology  with  One  Lagging  Interface 

The  author  assumes  a  simple,  three-component  system,  with  compo¬ 
nents  1,  2,  and  3  funded  at  the  6.7  —  operational  systems  development  level. 
MRL.  =  {7,7,7};  TRL.=  {7,7,7}  and  the  IRL  has  the  following  values:  IRL12  =  7, 
IRL13  =  7,  and  IRL23  =  4.  The  SRL  calculations  are  as  follows: 

mrl.  ={1,1,1}.  irl12  =  1,  irl13  =  1,  and  irl23  =  0.57,  irl1  =  mean[irZ12,  MJ  =  1 ,  irl2  = 
mean[irZ12,  irZ93]  =  0.79, ,  irl3  =  mean[irZ23,  irZJ  =  0.79. 

SRL.  =  mrl.  TRL.irl.  =  {1*7*1, 1*7*0.79,1*7*0.79}  =  {7, 5. 5, 5.5} 

SRL  =  mean[SRL.]  =  6 

Analysis 

SRL  =  6,  but  the  TRLs  are  all  at  7.  This  alerts  management  that  there  is 
at  least  one  interface  or  manufacturing  issue.  Examining  the  component 
SRLs  reveals  that  SRLX  =  TRL3  =  7,  but  the  other  two  SRLs  lag  TRL,  indi¬ 
cating  that  the  interfaces  from  component  2  to  3  are  lagging  and  should  be 
addressed  before  developing  the  component  technologies  further. 

Practical  Example— High  Energy  Laser  System 

Note:  This  is  an  example  and  not  representative  of  any  particular  sys¬ 
tem.  A  high  energy  laser  system  is  in  early  research  and  development, 
primarily  funded  by  6.2  and  6.3  sources.  It  is  composed  of  at  least  the  fol¬ 
lowing  subsystems:  laser,  beam  director  (BD),  thermal  management  (TM), 
electrical  management  (EM),  structural  support  (Struct),  atmospheric 
propagation  (Atmos),  target,  target  acquisition,  tracking,  pointing  (ATP), 
and  battle  management  and  controls  (BM).  A  TRL  assessment  might  be  as 
follows  (Table  4). 


[|  TABLE  4.  SAMPLE  TRL/MRL  RATINGS  j 
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Note  that  the  target  TRL  and  MRL  are  “n/a”  because  the  system  does  not 
involve  building  the  target,  but  the  atmosphere  and  ATP  form  an  interface 
with  the  target  so  an  IRL  is  associated  with  the  target,  but  no  TRL. 
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An  IRL  matrix  might  look  as  shown  in  Table  5,  if  most  effort  had  been  placed 
into  developing  the  laser,  beam  director,  ATP  algorithms;  and  target  infor¬ 
mation,  but  not  much  effort  placed  on  ‘system’  issues,  such  as  the  electrical 
or  thermal  management  systems  or  the  controls  architectures.  For  simplic¬ 
ity,  only  the  upper  half  of  the  matrix  is  shown  assuming  that  IRL..  =  IRL... 


TABLE  5.  SAMPLE  IRL  RATINGS 
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Not  every  component  has  every  kind  of  interface  so  that  the  n/a  values  in 
Table  5  are  simply  not  part  of  the  calculation.  Applying  the  equations  in 
Tables  1,  2,  and  3  yields  the  results  shown  in  Table  6,  assuming  normaliza¬ 
tion  by  the  6.3  fundingvalues  from  Table  1  and  Table  2.The  SRL  =  2.19.  The 
average  TRL  =  3.1. 


TABLE  6.  SAMPLE  IRL,  MRL  AND  SRL  COMPONENTS 
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The  average  SRL  is  below  the  average  TRL,  indicating  that  there  are  some 
integration  or  manufacturing  issues  that  should  be  addressed  before  the 
components  are  developed  further.  Specifically,  the  laser  subcomponent 
itself  is  a  TRL  4,  with  an  appropriate  level  of  manufacturability;  however, 
its  average  IRL  is  the  lowest  of  any  of  the  other  subsystems.  Such  a  system 
is  in  danger  of  developing  a  main  component  that  cannot  be  integrated, 
demonstrated  with  a  prototype  system  at  an  appropriate  level,  or  that 
will  come  up  with  extensive  integration  issues  late  in  development.  These 
integration  issues  may  prove  to  be  very  costly  and  time-consuming  to  fix. 
It  would  be  best  to  develop  the  laser-thermal,  laser-electrical,  laser-battle 
management,  and  laser  control  interfaces  before  continuing  to  mature  the 
laser  technology  itself.  The  side  benefit  would  be  the  ability  to  demonstrate 
early  prototype  laser  systems  rather  than  waiting  for  full  maturity  of  the 
final  laser  to  conduct  any  demonstrations,  which  would  be  conducive  to 
maintaining  the  interest  in  funding  this  technology  development  effort. 


Verbal  System  Readiness  Level 
Definitions 

The  proposed  mathematical  definition  of  SRL  permits  a  verbal  defini¬ 
tion  of  SRLs  in  a  way  that  the  Sauser  definition  and  mathematics  did  not. 
There  is  one  caveat  to  these  verbal  definitions:  they  strictly  hold  fast  at  those 
milestones  of  development  where  SRL  =  TRL.  It  is  possible  to  have  an  SRL 
of  3  with  TRLs  of  6  by  ignoring  interfaces  and  manufacturing,  in  which  case 
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the  following  definition  of  SRL  =  3  (Table  7)  would  not  be  accurate  because 
the  SRL  metric  is  significantly  lagging  the  TRL  metrics.  This  caveat  also 
helps  ensure  that  one  cannot  inappropriately  claim  a  high  level  of  SRL  by 
having  one  mature  component  mask  a  less  mature  one. 


TABLE  7.  SYSTEM  READINESS  LEVEL  DEFINITIONS  jj 

SRL 

Name 

Definition 

1 

System  concept 

The  system  concept  has  been  identified 
to  include  the  subsystems.  Overall  system 
functional  requirements  are  qualitatively 
understood. 

2 

System 

technologies 

Subsystem  technology  path  identified  to  include 
a  specific  technology  solution.  Technology, 
manufacturing,  and  interface  drivers  understood. 

3 

System  proof  of 
concept 

Experimental  evidence  has  been  obtained  that 
the  system  is  possible  in  principle  to  develop  and 
manufacture. 

4 

System 

component 

verification 

All  system  components  have  been  built 
and  tested  in  a  laboratory  environment 
separately.  Numerical  studies  show  component 
compatibility. 

5 

System 

component 

validation 

All  system  components  have  been  built  and 
tested  in  a  relevant  or  emulated  production  and 
deployment  environment.  Components  with 
simulated  interfaces  have  been  tested. 

6 

System 

prototype 

demonstration 

A  system  prototype  has  been  demonstrated  and 
fabricated  in  a  relevant  environment.  Interface 
control  has  been  demonstrated  traceable  to  a 
deployed  environment. 

7 

System 

operational 

demonstration 

An  integrated  system  prototype  has  been 
demonstrated  and  fabricated  in  an  operational  / 
manufacturing  environment. 

8 

Actual  system 
demonstration 

The  production  representative  system  has  been 
demonstrated  in  an  operational  environment. 

9 

Operational 

system 

Production  system  is  used,  demonstrated,  and 
maintained  in  an  operational  environment. 

Generalized  Integration  Readiness  Level  Definitions 

The  author  proposes  the  simplified  critical  item  lists  (Table  8)  for  the 
IRLs  (U.S.  Navy,  2009).  The  simplified  lists  allow  a  wider  application  to 
physical  rather  than  IT  systems,  and  focus  on  the  few  truly  critical  mile¬ 
stones  rather  than  many  contributing  factors.  See  U.S.  Navy  (2009,  p.  6)  for 
a  comparison. 
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TABLE  8.  SIMPLIFIED  INTEGRATION  READINESS  LEVEL  DEFINITIONS 

IRL  Name 

(Sauser  et  al., 
2010)  Definition 

Critical  Items 

Identification 

An  Interface 
between 
technologies  has 
been  identified 
with  sufficient 
detail  to  allow 
characterization  of 
the  relationship. 

There  exists  a  functional  flow  block 
diagram  for  the  technology  and 
its  interfaces  in  a  notional  system 
concept. 

Subject  matter  experts  for  each  of 
the  connecting  technologies  have 
been  identified  and  a  technical 
interchange  held. 

k) 

Characterization 

There  is  some 
level  of  specificity 
to  characterize 
the  Interaction 
(i.e.,  ability  to 
influence)  between 
technologies 
through  their 
interface. 

Input  and  output  parameters  have 
been  identified  for  each  interface. 

CaJ 

Compatibility 

There  is 

Compatibility 
(i.e.,  common 
language)  between 
technologies 
to  orderly  and 
efficiently 
integrate  and 
interact. 

Parametric  or  physics-based 
models  describe  the  interface  at  the 
qualitative  level  so  that  the  impact 
on  each  of  the  identified  parameters 
can  be  modeled  at  the  system  level. 
Interface  risks  have  been  identified. 
Interface  constraints  have  been 
identified. 

Quality  and 
Assurance 

There  is  sufficient 
detail  in  the 

Quality  and 

Assurance  of 
the  integration 
between 
technologies. 

A  solution  space  exists  to  meet 
design  concept  requirements. 

Generic  interface  models  have  been 
validated  by  experiment. 

o 

5  £ 

o 
u 

There  is  sufficient 
Control  between 
technologies 
necessary  to 
establish,  manage, 
and  terminate  the 
integration. 

Interfaces  are  well  defined. 

Interfaces  have  been  demonstrated 
in  a  laboratory  environment. 

Specific  interface  models  have  been 
validated  by  experiment. 
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TABLE  8, 

CONTINUED 

IRL  Name 

(Sauser  et  al., 

2010)  Definition 

Critical  Items 

CD 

Information 

The  integrating 
technologies  can 
Accept,  Translate, 
and  Structure 
Information  for 
their  intended 
application. 

Control  architecture  is  developed. 
Software  components  work 
together. 

Individual  modules  are  tested 
with  control  signals  to  verify 
performance. 

Integrated  system  demonstrations 
are  completed. 

c  The  integration 

o  s  of  technologies 

ro  ;§  has  been  Verified 

~  o  and  Validated  with 

>  sufficient  detail  to 

>  c  be  actionable, 
rc 


Fully  integrated  prototype  in 
simulated  operational  environment. 
Each  interface  tested  under  stressed 
and  anomalous  conditions. 


Actual  integration 

System  is  fully  integrated  in  an 

completed  and 

operational  environment. 

Mission  Qualified 

All  flight  and  safety  qualifications 

8 

C/l  — 

c />  ro 

through  test  and 

are  completed  for  all  technologies 

*3 

demonstration, 

and  interfaces. 

in  the  system 
environment. 

Form,  fit,  and  function  are  verified. 

Integration  is 

System  is  fully  integrated  and 

C  c 

Mission  Proven 

has  demonstrated  operational 

9 

'l/>  5 

through  successful 

effectiveness. 

iz  a 

mission  operations.  • 

Interface  failure  rates  are  fully 

characterized. 


Use  of  the  SRL  Metric 

Any  time  the  performance  or  behavior  of  a  complex  system  is  summa¬ 
rized  by  a  single  number,  there  is  inevitable  loss  of  information  and  the 
potential  for  false  indication.  SRL  and  IRL  have  a  subjective  component  to 
them,  as  do  TRL  and  MRL.  The  existence  of  the  SRL  metric  will  not  com¬ 
pletely  compensate  for  organizational  or  programmatic  pressure  to  advance 
technologies  prematurely  to  meet  budget  and  schedule.  It  will,  however,  fos¬ 
ter  an  awareness  of  the  cost  of  doing  so.  The  SRL  metric,  as  defined  herein, 
is  designed  to  be  an  indication  that  a  system  or  component  is  ready  for  the 
next  step  in  development  when  the  system  readiness  is  commensurate  with 
the  technology  readiness.  From  equations  (5)  and  (11),  where  SRL  =  TRL  at 
the  system  level  and  SRL.  =  TRL.  at  the  component  level,  advancement  is 
appropriate.  Since  interfaces  cannot  be  more  mature  than  their  component 
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technologies,  SRL  will  lag  TRL  at  each  step  of  development.  At  that  point, 
the  program  will  move  to  a  higher  funding,  maturity,  or  development  cat¬ 
egory;  the  normalization  factors  will  change;  and  SRL  will  once  again  lag 
TRL  as  shown  conceptually  in  Figure  7.  This  built-in  safeguard  will  reduce 
the  possibility  of  a  mature  subset  of  the  system  overshadowing  a  less  mature 
part  and  giving  false  indications  (Kujawski,  2010).  A  further  safeguard  can 
be  implemented  by  limiting  the  values  of  the  normalized  IRLs  and  MRLs 
( mrl  and  irl  from  Tables  1  and  2)  to  a  maximum  of  1.0,  further  ensuring  that 
one  mature  component  cannot  mask  a  less  mature  one.  The  principle  that 
advancement  to  the  next  level  of  funding  or  acquisition  should  not  occur 
until  the  system  readiness  is  commensurate  with  the  technology  readiness 
can  and  should  be  applied  at  the  system  level  (when  SRL  =  mean[TRL.])  and 
at  the  component  technology  level  (when  SRL.  =  TRL.). 


Conclusions 

This  article  has  proposed  a  modification  to  the  Sauser  mathematics  of 
IRL  and  SRL  that  allows  an  SRL  metric  that  gives  a  clear  indicator  of  when 
a  component  technology  or  system  is  ready  for  further  advancement  and 
allows  for  standard  verbal  definitions  of  SRL .  SRL  and  IRL  need  to  be  incor¬ 
porated  into  the  system  engineering  process  early  in  development.  TRL  has 
been  a  valuable  metric;  however,  its  lack  of  emphasis  on  systems  issues  has 
resulted  in  divergent  development,  where  some  system  components  are 
developed  beyond  their  interfaces  and  manufacturing,  resulting  in  legacy 
decisions  that  impede  demonstration  and  integration.  A  useful  SRL  metric 
can  help  to  foster  more  balanced  and  cost-effective  technology  development. 
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Appendix 

Standard  Technology  Readiness  Level  and  Manufacturing 
Readiness  Level  Definitions 


|j  TABLE  A-1.  STANDARD  TRL  DEFINITIONS  f 

TRL 

Definition 

1 

Basic  principles  observed  and  reported 

2 

Technology  concept  and/or  application  formulated 

3 

Analytical  and  experimental  critical  function  and/or  characteristic 
proof  of  concept 

4 

Component  and/or  breadboard  validation  in  a  laboratory 
environment 

5 

Component  and/or  breadboard  validation  in  a  relevant  environment 

6 

System/subsystem  model  or  prototype  demonstration  in  a  relevant 
environment 

7 

System  prototype  demonstration  in  an  operational  environment 

8 

Actual  system  completed  and  qualified  through  test  and 
demonstration 

9 

Actual  system  proven  through  successful  mission  operations 

Note.  (DoD,  2011) 
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TABLE  A-2.  STANDARD  MRL  DEFINITIONS 

MRL 

Definition 

1 

Basic  manufacturing  implications  identified 

2 

Manufacturing  concepts  identified 

3 

Manufacturing  proof  of  concept  developed 

4 

Capability  to  produce  the  technology  in  a  laboratory  environment 

5 

Capability  to  produce  prototype  components  in  a  production¬ 
relevant  environment 

6 

Capability  to  produce  a  prototype  system  or  subsystem  in  a 
production-relevant  environment 

7 

Capability  to  produce  systems,  subsystems,  or  components  in  a 
production-representative  environment 

8 

Pilot  line  capability  demonstrated;  ready  to  begin  Low  Rate  Initial 
Production 

9 

Low  Rate  Initial  Production  demonstrated;  capability  in  place  to 
begin  Full  Rate  Production 

10 

Full  Rate  Production  demonstrated  and  Lean  production  practices 
in  place 

Note.  (DoD,  2012) 

TABLE  A-3.  DoD  STANDARD  FUNDING  CATEGORIES 

6.1 

Basic  Research 

6.2 

Applied  Research 

6.3 

Advanced  Technology  Development 

6.4 

Advanced  Component  Development  and  Prototypes 

6.5 

System  Development  and  Demonstration 

6.6 

RDT&E  Management  Support 

6.7 

Operational  Systems  Development 

Note.  (DAU,  2016) 
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Systems  engineers  are  faced  with  the  difficult  challenge  of  adhering  to 
broad  systems  engineering  (SE)  policies,  while  simultaneously  tailoring 
SE  processes  to  meet  the  unique  challenges  facing  their  projects.  Tailoring 
is  often  performed  in  an  ad  hoc  manner.  Determining  which  stages,  steps, 
and  artifacts  of  the  process  are  necessary  can  be  time-consuming  and  chal¬ 
lenging.  SE  guidebooks  across  industry  and  government  organizations  often 
stress  the  importance  of  tailoring,  yet  offer  little  practical  guidance  on  how 
to  perform  the  function.  This  article  proposes  a  model  for  automating  the 
SE  tailoring  process  through  the  definition  of  an  organizational  rule 
set  and  a  minimal  set  of  project-specific  inputs.  The  model  is  then 
analyzed  through  several  case  studies  within  the  Department  of 
Homeland  Security  to  evaluate  the  proposed  approach. 
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Most  organizations  develop  their  own  systems  engineering  (SE)  guide¬ 
lines  to  specify  the  stages,  reviews,  and  artifacts  that  should  be  executed 
on  projects.  Standard  processes  are  important  for  today’s  projects,  as  they 
employ  standardized  terminology  and  may  prevent  project  teams  from 
reinventing  the  wheel;  however,  formal  and  informal  processes  must  be 
balanced  to  yield  the  efficiencies  gained  through  standardization  and  the 
effectiveness  gained  by  adaptability  (Laufer,  2001,  p.  27).  Since  no  two 
projects  are  identical,  no  two  processes  are  either  (Humphrey,  1989,  p.  241). 
Organizations  must  develop  tailorable  guidelines  that  are  broad  enough  to 
span  the  range  of  projects  within  their  portfolio  (Hwang  &  Park,  2006,  p. 
37).  This  places  the  burden  of  tailoring  the  organization’s  SE  guideline  on 
project  systems  engineers.  This  article  proposes  a  rule-based  approach  to 
partially  automating  this  activity,  thereby  reducing  the  manual  burden  yet 
still  offering  a  significant  number  of  custom  SE  process  variations  to  best 
meet  a  project’s  needs. 


Process  Tailoring 


Process  tailoring  refers  to  the  modification  of  a  standardized  pro¬ 
cess  to  meet  the  unique  needs  of  a  project  (Xu  &  Ramesh,  2007,  p.  293). 
This  can  include  determining  which  stages  and  reviews  are  necessary 
and  should  be  executed  multiple  times,  which  artifacts  should  be  pro¬ 
duced,  or  even  what  content  within  an  artifact  is  necessary.  Successful 
tailoring  results  in  modified  processes  that  achieve  the  goals  of  the  stan¬ 
dard  process  model  (International  Organization 
for  Standardization/International 
Electrotechnical  Commission/Institute 
of  Electrical  and  Electronics  Engineers 
[ISO/IEC/IEEE]  15288,  2015,  p. 
86).  Project  teams  often  perform 


tailoring  in  an  ad  hoc  man¬ 
ner  without  guidelines  or  rules 
(Pedreira,  Piattini,  Luaces,  & 
Brisaboa,  2007,  p. 
1),  and  usually 
complete  the 
activity  based 
on  declarative 
memory  (Xu  &  Ramesh,  2009,  p.  282).  Process 
implementation  directly  affects  project  budget  and 
schedule,  as  well  as  product  quality.  Choosing  and  adapting 
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the  right  design  method  is  critical  to  successfully  executing  SE  concepts 
and  principles  (Verma  &  Fabrycky,  1997,  p.  592).  Systems  engineers  are  thus 
faced  with  the  difficult  challenges  of  adhering  to  broad  policies. 

While  research  related  to  process  tailoring  in  software  engineering  is  preva¬ 
lent,  similar  research  in  SE  is  limited.  Existing  SE  guidelines  and  handbooks 
stress  the  importance  of  tailoring  SE  processes  to  meet  the  unique  needs  of 
a  project,  but  provide  little  guidance  on  process  tailoring  (Browning,  Fricke, 
&  Negele,  2006,  p.  119;  Pereira  et  al.,  2007,  p.  72).  Fortune  and  Valerdi  (2013) 
propose  a  framework  for  reusing  SE  products  within  an  organization,  and 
highlight  key  considerations  for  process  reuse.  Among  them.  Fortune  and 
Valerdi  (2013,  p.  310)  argue  that  a  robust  knowledge  management  system  is 
critical  to  the  success  of  process  reuse.  They  also  note  the  reuse  of  a  product 
at  too  high  a  level  may  not  apply  to  the  specific  environment  of  the  new  proj¬ 
ect.  On  the  other  hand,  reusing  products  at  too  low  a  level  can  cause  tailoring 
of  enough  significance  that  the  effort  saved  by  process  reuse  may  be  lost 
(Fortune  &  Valerdi,  2013,  p.  305). 

Process  tailoring  can  be  considered  as  a  form  of  standard  process  reuse 
(Yoon,  Min,  &  Bae,  2001,  p.  202);  however,  tailoring  involves  process  appli¬ 
cation  to  a  unique  project  environment,  as  opposed  to  the  flexibility  of  the 
standard  process  itself.  Particularly  in  the  field  of  software  engineering, 
patterns  have  been  explored  as  a  viable  option  for  process  reuse;  in  fact, 
Cloutier  and  Verma  (2007)  extend  the  concept  to  a  method  for  developing 
pattern  forms  in  systems  architecture.  In  addition,  Hagge  and  Lappe  (2005, 
p.  24)  emphasize  the  need  for  capturing  knowledge  that  can  be  reused  in 
future  projects.  They  propose  four  introductory  patterns  for  requirements 
engineering  and  discuss  a  method  for  sharing  observations  across  projects 
(Hagge  &  Lappe,  2005). 


Method  Engineering 

Extending  knowledge  from  the  software  process  domain  to  the  SE 
process  domain  has  been  ongoing  for  quite  some  time  (Boehm,  2006,  p. 
5).  Research  in  software  process  tailoring  is  well-documented  and  can  be 
extended  to  SE.  Henderson-Sellers,  Ralyte,  Agerfalk,  and  Rossi  (2014)  pro¬ 
vide  an  extensive  literature  review  of  the  application  of  Method  Engineering 
(ME)  to  specific  situations  in  software  development.  Their  review  spans  the 
various  approaches  to  method  tailoring  and  case  studies  in  the  application  of 
method  tailoring.  Kuhrmann,  Mendez  Fernandez,  and  Tiessler  (2014)  also 
conducted  a  thorough  literature  review  of  ME,  which  pertains  to  the  process 
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of  constructing  methods  for  software  development.  They  highlight  83  arti¬ 
cles  contributing  to  the  research  of  software  process  tailoring,  and  conclude 
that  “strong  empirical  evidence  of  the  feasibility  of  ME”  is  still  lacking. 

Software  Process  Tailoring  Approaches 

A  wide  variety  of  approaches  to  software  process  tailoring  have  been 
documented  in  literature.  The  case  retrieval-based  method  formulates  a  tai¬ 
lored  software  process  based  on  retrieving  a  tailored  process  from  a  library 
of  historical  projects  (Ahn,  Ahn,  &  Park,  2003;  Park  &  Bae,  2013).  Retrieval 
is  performed  by  comparing  values  of  domain  factors  of  a  new  project  to  those 
of  past  projects  in  the  library.  Two  primary  challenges  exist  with  the  case 
retrieval  approach.  First,  the  extensiveness  of  the  library  of  previous  methods, 
which  requires  a  mature  organization,  limits  the  accuracy  of  the  case  retrieval 
approach.  And  second,  the  organization  must  have  an  established  knowledge 
management  system  that  not  only  archives  previous  pr oj  ects,  but  also  tags  the 
projects  with  the  values  and  domain  factors  required  for  retrieval. 

In  the  preformed  approach,  an  organization  develops  a  standard  set  of  tai¬ 
lored  processes  that  apply  to  particular  categories  of  projects  (Plogert,  1996). 
The  challenge  with  this  approach  is  that  it  only  includes  the  first  level  of 
tailoring,  such  as  hardware  or  software,  and  does  not  address  the  significant 
variability  that  can  exist  from  project  to  project  within  an  organization. 
In  addition,  this  approach  does  not  easily  allow  for  growth;  to  modify  the 
approach  based  on  lessons  learned  would  require  revisiting  the  pretailored 
processes  altogether. 

Hausen  (1998)  employs  a  rule-based  approach  to  software  process  tailor¬ 
ing.  The  rule-based  approach  identifies  values  of  factors  that  address  the 
rationale  for  tailoring  based  on  specific  project  characteristics.  An  engineer 
creates  each  rule  such  that  for  a  specific  value  of  a  domain  factor  that  exists 
for  a  new  project,  certain  process  elements  are  tailored  (Kang,  Song,  Park, 
Bae,  Kim,  &  Lee,  2008,  p.  54).  Kang  et  al.  (2008)  goes  further  by  combining 
the  rule-based  approach  with  case  retrieval  where  rules  are  used  to  retrieve 
the  case  most  similar  to  the  attributes  of  the  new  project. 

Jaufman  and  Miinch  (2005,  pp.  328-342)  used  both  top-down  and  bot- 
tom-up  approaches  to  software  process  tailoring,  allowing  for  increased 
efficiency  and  process  adherence,  respectively.  This  approach  leverages 
both  the  preformed  and  rule-based  methodologies. 

Hanssen,  Westerheim,  and  Bjornson  (2005)  use  brainstorming  to  tailor  a 
software  process  in  an  organization.  Baldassarre,  Caivano,  Visaggio,  and 
Visaggio  (2002)  propose  the  use  of  patterns  for  software  process  tailoring. 
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As  previously  stated,  a  significant  amount  of  research  exists  in  process  tai¬ 
loring  for  the  software  engineering  field.  While  similar  research  exists  in 
SE  process  tailoring,  the  opportunity  exists  to  apply  software  engineering 
research  to  SE. 

By  proposing  a  model  that  simplifies  SE  process  tailoring,  this  article  seeks 
to  bridge  the  gap  in  SE  literature  between  SE  process  research  that  stresses 
the  importance  of  tailoring  yet  offers  little  guidance  on  howto  perform  the 
function,  and  the  practitioners  who  have  to  perform  SE  process  tailoring  to 
meet  the  specific  needs  of  their  projects.  The  proposed  rule-based  approach 
to  SE  process  tailoring  offers  a  significant  number  of  custom  SE  process 
variations  as  compared  to  the  preformed  approach  and  without  the  reliance 
on  a  deep  repository  required  of  a  case-retrieval  approach.  With  this  pro¬ 
posed  approach,  the  number  of  tailoring  decisions  could  be  reduced  from  the 
total  number  of  elements  in  a  governing  SE  process  to  the  number  of  rules 
established  within  an  SE  organization  and  applied  to  the  project. 

The  next  section  of  this  article  proposes  the  SE  Process  Tailoring  Model 
(SEPTM),  developed  starting  with  tailoring  considerations  identified  in  the 
INCOSE  Systems  Engineering  Handbook,  published  by  the  International 
Council  on  Systems  Engineering  (INCOSE,  2015),  and  that  can  be  used  to 
generate  a  project-specific  SE  process  based  on  a  project’s  unique  charac¬ 
teristics  and  environment.  The  discussion  on  the  SEPTM  is  then  followed 
by  a  section  that  introduces  the  analysis  of  24  case  studies  within  the 
Department  of  Homeland  Security  (DHS),  thereby  evaluating  the  model 
against  real  world  SE  process  tailoring  instances.  Finally,  the  article  high¬ 
lights  conclusions  of  the  research  and  areas  of  future  work. 
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Systems  Engineering  Process  Tailoring 
Model  (SEPTM) 

Figure  1  presents  an  overview  of  the  proposed  SEPTM.  At  a  high  level, 
the  authors  constructed  the  model  based  on  an  analysis  of  the  generic  SE 
process  described  in  the  INCOSE  Systems  Engineering  Handbook  that 
influences  process  tailoring.  This  analysis  forms  the  basis  of  an  initial  set 
of  Tailoring  Considerations  (TC).  From  the  initial  set  of  TCs,  a  subset  is 
then  selected  that  is  applicable  to  a  particular  organization.  A  rule  set  is 
developed  that  links  the  TCs  and  the  organizational  SE  process.  A  systems 
engineer  can  then  apply  the  rule  set  to  the  unique  attributes  of  a  project  to 
generate  a  customized  SE  process  based  on  the  conditions  of  a  particular 
project.  The  following  subsections  further  describe  each  of  these  activities. 


FIGURE  1.  OVERVIEW  OF  THE  SYSTEMS  ENGINEERING  PROCESS 
TAILORING  MODEL  (SEPTM) 


j  Step  1 
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An  important  aspect  of  the  model  is  determining  the  TCs  used  to  iden¬ 
tify  critical  aspects  of  a  project.  The  TCs  will  broadly  inform  what  SE 
activities  are  needed  for  a  given  project.  The  development  of  the  initial 
organizational  TC  list  is  based  on  an  analysis  of  the  general  SE  process 
described  in  the  INCOSE  Systems  Engineering  Handbook  and  its  discussion 
of  considerations  and  enablers  for  process  tailoring  (INCOSE,  2015).  The 
handbook  states  the  appropriate  approach  will  vary  based  on  the  unique 
needs  of  a  project.  It  also  describes  various  life-cycle  approaches,  including 
the  waterfall,  incremental,  and  agile  approaches.  In  the  waterfall  approach, 
the  project  team  performs  SE  stages  serially;  in  the  incremental  approach, 
the  project  team  performs  SE  stages  serially  multiple  times;  and  in  the  agile 
approach,  the  project  team  iterates  through  short  development  cycles  with 
frequent  product  releases  (Aoyama,  1998).  As  part  of  the  agile  development 
cycles,  project  teams  perform  the  SE  stages  of  requirements  definition, 
design,  development,  and  testing  in  an  iterative  fashion.  Moreover,  project 
teams  perform  and  update  the  associated  reviews  and  documentation  with 
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each  release  (Cao  &  Ramesh,  2008,  p.  63).  Each  of  these  approaches  varies  in 
terms  of  stage  and  review  frequency,  as  well  as  the  amount  of  documentation 
required.  As  a  result,  a  project’s  life-cycle  approach  must  be  a  critical  TC. 

The  INCOSE  Systems  Engineering  Handbook  also  describes  cross-cutting 
technical  methods  and  includes  a  section  on  specialty  engineering  analyses 
that  must  be  considered  by  a  project’s  systems  engineer;  however,  not  all 
of  the  analyses  will  apply  to  every  project  (INCOSE,  2015).  For  example, 
interoperability  should  be  considered  for  projects  that  interface  with  other 
projects.  An  organization  may  have  specific  artifacts  that  are  required  to 
address  interoperability  for  certain  projects  or  allow  for  deleting  interop¬ 
erability  artifacts  based  on  the  project  type. 

The  handbook’s  tailoring  section  includes  a  discussion  of  SE  process  tailor¬ 
ing  and  identifies  considerations  and  enablers  for  tailoring:  project  scope, 
risk  tolerance,  complexity  and  precedence  of  the  system,  and  organiza¬ 
tional/enterprise  policies  and  infrastructure  (INCOSE,  2015).  While  the 
handbook  identifies  risk  tolerance  as  a  factor  in  process  tailoring,  it  does 
advise  of  the  importance  of  placing  tailoring  emphasis  on  the  system  and 
the  project  objectives,  or  project  scope. 

Analysis  of  the  general  SE  process  and  tailoring  discussion  in  the  INCOSE 
Systems  Engineering  Handbook  resulted  in  the  following  list  of  TCs: 


TAILORING  CONSIDERATIONS 


Life-cycle  Approach 
Project  Scope 

Complexity  and  Precedence  of  the 
System 

Organizational/Enterprise  Policies 
and  Infrastructure 

Modeling,  Simulation,  and 
Prototyping 

Affordability/Cost  Effectiveness/ 
Life-Cycle  Cost  Analysis 

Electromagnetic  Compatibility 
Analysis 

Environmental  Engineering/Impact 
Analysis 

Interoperability  Analysis 


Logistics  Engineering  Analysis 

Manufacturing  and  Producibility 
Analysis 

Mass  Properties  Engineering 
Analysis 

Reliability,  Availability,  and 
Maintainability 

Resilience  Engineering 
System  Safety  Analysis 
Systems  Security  Engineering 
Training  Needs  Analysis 

Usability  Analysis/Human  Systems 
Integration 

Value  Engineering 


Defense  ARJ,  July  2016,  Vol.  23 No.  3: 274-297 


281 


A  Publication  of  the  Defense  Acquisition  University 


http://www.dau.mil 


Of  the  19  TCs,  the  first  four— Life-cycle  Approach,  Project  Scope, 
Complexity  and  Precedence  of  the  System,  and  Organizational/Enterprise 
Policies  and  Infrastructure— apply  to  all  organizations,  as  the  INCOSE 
Systems  Engineering  Handbook  suggests.  The  remaining  15  vary  in  appli¬ 
cability,  depending  on  the  types  of  projects  an  organization  executes.  This 
requires  an  organization  to  determine  the  appropriate  subset  of  TCs  for 
their  project  portfolio. 

Refine  the  Set  of  Tailoring  Considerations  for  an  Organization 

In  the  second  analytical  step,  the  overarching  TC  list  is  reduced  to 
one  that  includes  only  the  TCs  relevant  for  that  organization.  This  step  is 
performed  by  comparing  the  TC  list  to  organizational  policies  and  the  orga¬ 
nizational  SE  process.  For  each  TC,  possible  values  are  identified.  The  TCs 
can  have  values  that  are  either  yes/no  (e.g.,  environmental  impacts  need  to 
be  considered  or  not)  or  multiple  (e.g.,  life-cycle  approach  maybe  waterfall, 
incremental,  or  agile).  The  case  study  in  the  next  section  of  this  article  offers 
a  more  detailed  discussion  of  how  this  step  is  performed. 

Define  Tailoring  Conditions 

Before  a  tailoring  rule  set  can  be  developed,  some  tailoring  operations  must 
be  defined.  For  the  purposes  of  this  research,  the  following  definitions  apply: 

•  “Standard”  is  used  when  a  stage,  review,  or  artifact  is  to  be 
developed  consistent  with  the  scope  of  the  activity  defined  in 
the  organizational  SE  policy. 

•  “Tailored”  is  used  when  a  stage,  review,  or  artifact  is  modified 
from  the  scope  in  the  organizational  SE  policy.  This  could 
include  combining  two  or  more  stages,  reviews,  or  artifacts 
into  one.  This  could  also  include  repeating  stages  or  reviews 
multiple  times  within  a  project.  For  example,  the  process  tai¬ 
loring  model  recommends  that  the  Critical  Design  Review 
occur  multiple  times  in  a  project  that  uses  incremental  devel¬ 
opment  methodology. 

•  “Deleted”  is  used  when  the  stage,  review,  or  artifact  is  not 
needed  and  removed  from  the  project’s  tailored  SE  process 
altogether.  An  example  of  this  is  the  removal  of  a  System  Design 
Document  for  a  Commercial  Off-The-Shelf  (COTS)  acquisition. 
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Establishing  the  Rule  Set 

Once  the  TCs  have  been  selected  for  an  organization  and  tailoring  con¬ 
ditions  defined,  a  rule  set  must  be  generated  to  link  the  TCs  and  associated 
values  to  the  elements  of  the  organization’s  standard  SE  process.  Process 
elements  are  those  SE  tasks  and  activities  identified  within  the  organiza¬ 
tion’s  SE  process.  Table  1  shows  the  matrix  used  to  establish  the  rule  set.  In 
area  1  of  the  table,  all  of  the  elements  (1  to  M)  of  the  standard  SE  process  are 
listed  vertically.  For  example,  the  DHS  SE  process  has  107  process  elements 
(“M”).  In  area  2,  each  TC  (lto  N)  and  associated  values  (lto  O)  are  listed  hor¬ 
izontally.  “N”  represents  the  number  of  TCs  identified  for  the  organization, 
and  the  values  represent  potential  attributes  of  a  project  within  the  orga¬ 
nization.  As  an  example,  if  TC  1  is  Interoperability  Analysis,  the  potential 
values  would  be  “yes”  and  “no”  given  that  a  system  may  be  stand-alone  or 
connected.  For  each  intersection  of  a  process  element  and  a  TC-value  com¬ 
bination  in  area  3,  a  tailoring  rule  is  established.  This  requires  two  levels 
of  decisions.  First,  a  determination  is  made  as  to  whether  a  TC  is  relevant 
or  not  for  a  particular  process  element;  and  second,  if  so,  a  determination  is 
made  for  each  value  of  the  TC  whether  the  process  element  should  be  kept 
standard,  tailored,  or  deleted.  Thus,  each  relevant  intersection  in  area  3 
will  contain  either  “S”  for  standard,  “T”  for  tailored,  or  “D”  for  deleted.  In 
summary,  the  resulting  rule  set  is  a  list  of  TCs,  for  which  each  value  drives  a 
tailoring  decision  (standard,  tailored,  or  deleted)  for  every  relevant  process 
element  within  the  organizational  SE  process. 


TABLE  1.  ESTABLISHING  THE  RULE  SET 
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Considerations 
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Apply  Model  to  a  New  Project  and  Validate 

Once  the  model  has  been  developed  for  an  organization,  project  systems 
engineers  can  use  it  for  tailoring  SE  processes  to  meet  the  needs  of  a  specific 
project.  To  do  so,  a  systems  engineer  must  have  a  solid  understanding  of 
the  acquisition  strategy  and  the  project  environment.  This  is  true  whether 
applying  a  rule-based  methodology  as  is  the  case  here,  or  tailoring  an  SE 
process  in  an  ad  hoc  fashion.  To  exercise  the  model,  a  systems  engineer 
must  align  characteristics  of  the  project  to  a  value  associated  with  each  TC. 
Once  this  is  complete,  the  model  outputs  the  tailored  SE  process,  identifying 
which  of  the  full  set  of  the  organization’s  SE  process  elements  should  be 
executed  as  standard,  which  should  be  tailored,  and  which  can  be  deleted. 
After  the  model  has  been  applied  to  a  real-world  situation,  the  organiza¬ 
tion  must  incorporate  lessons  learned  back  into  the  model  by  reassessing 
the  organization’s  list  of  TCs  and  the  organization’s  rule  set.  Moreover,  as 
SE  research  is  constantly  refining  best  practices  and  industry  standards 
are  updated,  the  organization  must  also  revisit  the  initial  TC  set  to  assess 
changes  from  industry. 


Case  Study 

The  objective  of  this  research  is  to  use  project  attributes  and  a  rule- 
based  approach  to  reduce  the  amount  of  manual  tailoring  of  the  SE  life 
cycle.  Henderson-Sellers  et  al.  (2014,  p.  169)  note  that  determining  the  best 
project  methodology  requires  specific  tailoring  to  a  project’s  unique  situ¬ 
ation.  Measuring  the  benefit  of  such  an  activity  can  be  based  on  the  amount 
of  effort  required  to  tailor  standard  processes  (Xu  &  Ramesh,  2007).  For 
purposes  of  this  analysis,  this  article  extends  that  definition  to  the  number 
of  manual  decisions  required  to  produce  a  project-specific  SE  process.  By 
design,  this  model  fixes  the  number  of  decisions  on  the  number  of  TCs 
established  for  an  organization.  If  an  organization  applies  all  of  the  initial 
19  TCs,  the  number  of  manual  decisions  will  be  19.  The  DHS,  an  organiza¬ 
tion  with  a  large  acquisition  portfolio,  has  an  SE  process  consisting  of  107 
process  elements,  resulting  in  107  associated  process  tailoring  decisions 
(DHS,  2010).  Provided  the  model  can  reproduce  the  tailoring  decisions 
made  within  the  DHS  case  studies,  the  potential  benefit  of  this  proposed 
approach  to  SE  process  tailoring  can  be  demonstrated  by  reducing  manual 
decisions  from  107  to  10,  which  as  described  in  the  following  subsections, 
is  the  number  of  TCs  determined  for  the  DHS  case  study.  As  such,  the  fol¬ 
lowing  subsections  focus  on  the  accuracy  of  the  model  in  comparison  to 
case  studies. 
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The  objective  of  this  research  is  to  use  project  attri¬ 
butes  and  a  rule-based  approach  to  reduce  the  amount 
of  manual  tailoring  of  the  SE  life  cycle. 


Case  Study  Data 

To  perform  the  case  study  analysis,  the  authors  required  two  key  ele¬ 
ments.  First,  an  organization’s  SE  policy  was  required  to  serve  as  the  process 
baseline.  The  baseline  was  used  to  establish  the  linkages  between  the  over¬ 
arching  TCs  and  the  process  elements  contained  within  the  baseline.  The 
data  source  for  this  research  was  the  DHS  Systems  Engineering  Lifecycle 
Process  Guide  (DHS,  2010).  The  DHS  SE  process  identifies  nine  SE  stages, 
11  technical  reviews,  and  87  artifacts  to  be  executed  through  the  life  of  a 
project,  for  a  total  of  107  SE  process  elements. 

The  second  key  element  required  is  a  set  of  tailored  SE  processes  imple¬ 
mented  on  projects  within  the  organization.  The  authors  selected  24  DHS 
projects  for  analysis  based  on  data  availability  and  variability  across  the 
TCs  (Table  2).  For  example,  some  projects  executed  full  research  and  devel¬ 
opment,  while  others  were  COTS  acquisitions;  some  executed  a  waterfall 
development  approach,  and  others,  incremental  or  agile.  The  authors  gath¬ 
ered  and  analyzed  quantitative  data  regarding  tailoring  outcomes  from  each 
of  the  24  projects  and  compared  actual  and  modeled  tailoring  outcomes. 


TABLE  2.  CASE  STUDY  PROJECT  VARIABILITY 

Case  Study 

Tailoring  Considerations 

Possible  Values 

#  Projects 

Security  Impact 

Impact/No  Impact 

17/7 

Privacy  Impact 

Impact/No  Impact 

9/15 

Intelligence  Impact 

Impact/No  Impact 

3/21 

Interoperability  Impact 

Impact/No  Impact 

11/13 

Accessibility  Impact 

Impact/No  Impact 

13/11 

Technology  Demonstration 
Planned 

Planned/Not  Planned 

7/17 

Environmental  Impact 

Impact/No  Impact 

7/17 

Development  Methodology 

Waterfall/  Incremental/Agile 

11/11/2 

Development  Type 

Development/COTS 

19/6 

Project  Size 

<  $300M,  >  $300M 

6/18 
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Selection  of  Organizational  Tailoring  Considerations 

The  first  step  in  applying  the  tailoring  model  to  an  organization  is 
to  determine  the  subset  of  the  19  overarching  TCs  that  are  applicable 
to  the  organization.  As  stated  previously,  the  first  four  TCs  apply  to  all 
organizations  and  were  adapted  for  the  DHS  TC  list.  Analysis  of  the  DHS 
organizational/enterprise  policies  yielded  additional  considerations  per¬ 
taining  to  privacy  and  intelligence.  Additional  analysis  of  the  DHS  SE 
process  was  performed  to  determine  which  of  the  remaining  15  are  appli¬ 
cable.  Of  the  15,  five  were  applicable  to  all  projects  per  the  DHS  SE  process, 
five  were  not  discussed  in  the  DHS  SE  process  guide,  and  five  may  apply 
depending  on  the  needs  of  the  specific  project.  The  five  TCs  that  may  be 
applicable  were  included  as  organizational  TCs  in  the  model.  Asa  result,  the 
10  organizational  TCs  identified  and  applied  for  this  research  are  as  follows: 

1.  Security  Impact 

2.  Privacy  Impact  (Organizational/Enterprise  Policies  and  Infrastructure) 

3.  Intelligence  Impact 

4.  Interoperability  Impact 

5.  Accessibility  Impact  (Usability) 

6.  Technology  Demonstration  (Modeling,  Simulation,  and  Prototyping) 

7.  Environmental  Impact 

8.  Development  Methodology  (Life-cycle  Approach) 

9.  Development  Type  (Complexity  and  Precedence  of  the  System) 

10.  Project  Size  (Project  Scope) 

With  the  10  organizational  TCs  in  place,  the  rule  set  for  DHS  was  established 
by  linking  the  TCs  to  the  standard  DHS  SE  process  as  depicted  in  Table  3. 
Note  that  for  any  given  TC,  only  a  subset  of  process  elements  is  relevant;  that 
is,  empty  table  entries  reflect  that  there  is  no  relevance  between  that  TC  and 
the  intersecting  standard  SE  process  element.  For  example,  in  Table  3,  four 
TCs  (2,  7,  9,  and  10)  are  relevant  to  process  element  No.  1. 
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Case  Study  Projects 

With  the  rule  set  established  for  DHS  as  described  previously,  the  next 
step  was  to  input  the  attributes  of  each  of  the  24  projects  into  the  model  to 
generate  a  custom  SE  process  for  each  project.  Figure  2  depicts  an  example 
project’s  input  and  output,  highlighting  that  the  input  is  addressing  the  10 
TCs  based  on  pr  oj  ect  characteristics,  and  the  output  is  a  tailoring  decision 
for  each  of  the  107  elements  within  the  DHS  SE  process. 


FIGURE  2.  EXAMPLE  MODEL  INPUT  AND  OUTPUT  FOR  A  PROJECT 


Model  Input  for  Project  1 

1.  Security  Impact 
OYes  *No 

2.  Privacy  Impact 
•Yes  ONo 

3.  Intelligence  Impact 
OYes  *No 

4.  Interoperability  Impact 
•Yes  ONo 

5.  Accessibility  Impact 
•Yes  ONo 

6.  Technology  Demonstration 
•Yes  ONo 

7.  Environmental  Impact 
OYes  *No 

8.  Development  Methodology 
•Waterfall  Olncremental  OAgile 

9.  Development  Type 
OFull  Scale  •COTS/NDI 

10.  Program  Size 
0>$300M  •<  $300M 


Model  Input  for  Project  1 

DHS  Process  Element 

Tailoring  Decision 

1  S 

D 

3 

S 

< 

S 

5 

S 

:  : 

103 

D 

104 

T 

105 

T 

106 

D 

107 

T 

Note.  COTS/NDI  =  Commercial  Off-The-Shelf/Nondevelopmental  Item. 


Three  questions  formed  the  analysis  of  each  TC.  Table  4  lists  those  ques¬ 
tions  and  provides  a  corresponding  example  using  the  Security  TC.  The 
authors  assessed  each  of  the  10  TCs  by  comparing  the  model  output  for  each 
project  to  the  data  from  each  project’s  SE  process  tailoring  plan.  A  default 
process  was  established  based  on  default  project  attributes  within  the  rule 
set.  Projects  that  had  an  attribute  different  from  the  default  were  compared 
to  the  model  for  that  particular  attribute.  For  example,  the  default  project 
attribute  for  the  Security  TC  is  “No”;  therefore,  the  14  projects  that  do  have 
security  impacts  were  compared  to  the  model  to  assess  whether  the  projects 
performed  the  process  elements  relevant  to  that  TC. 
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TABLE  4.  CASE  STUDY  ANALYSIS  QUESTIONS 


Analysis  Question  Security  TC  Example 


For  a  given  pairing  of  process 
element  and  tailoring  consideration, 
does  the  tailoring  decision 
generated  by  the  model  for  a 
project  match  the  tailoring  decision 
actually  executed  on  the  project? 

For  the  given  pairing  of  process 
element  and  tailoring  consideration, 
what  percentage  of  matches 
between  the  project  data  and  their 
corresponding  model  outputs 
exists? 

What  percentage  of  process 
elements  have  at  least  75% 
matching  between  model  output 
and  actual  project  data? 


For  a  project  with  security  impacts, 
did  the  model  and  project  data 
both  include  a  plan  to  execute  the 
disaster  recovery  plan  process 
element? 

For  the  14  projects  that  do  have 
security  impacts,  what  percentage 
of  the  projects  included  a  plan 
to  execute  the  disaster  recovery 
plan  process  element  as  the  model 
suggests? 

What  percentage  of  the  15  process 
elements  relevant  to  the  Security  TC 
have  a  75%  match  between  model 
output  and  actual  project  data? 


Table  5  summarizes  the  case  study  analysis  approach  using  the  Security  TC 
as  an  example.  Questions  listed  in  Table  4  trace  to  Table  5  as  “Ql”,  “Q2”,  and 
“Q3”.  Column  1  represents  the  process  elements  relevant  to  the  Security  TC. 
Column  2  shows  the  tailoring  decision  of  the  model  for  the  Security  TC  and 
relevant  process  elements;  the  next  set  of  columns  shows  the  pertinent  data 
from  the  24  projects,  and  an  assessment  of  whether  or  not  each  matches  the 
model  (Ql).  The  last  column  in  the  table  calculates  the  percentage  of  proj¬ 
ects  that  match  the  model  for  each  process  element  (Q2).  Consistent  with 
validation  studies  of  software  engineering  practices,  a  75  percent  acceptable 
matching  level  is  used  (Daneva  &  Ahituv,  2010,  p.  282;  Krishnan  &  Kellner, 
1999,  p.  806;  Ramasubbu,  Krishnan,  &  Kompalli,  2005,  p.  83).  Thus,  for  a  TC 
to  be  considered  valid,  the  model  output  for  each  relevant  process  element 
must  match  at  least  75  percent  of  the  relevant  projects  (last  column  in  Table 
5),  and  at  least  75  percent  of  the  process  elements  linked  to  that  TC  must 
meet  that  threshold  (Q3).  This  process  described  for  the  Security  TC  was 
repeated  for  each  of  the  10  TCs. 
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1  TABLE  5.  CASE  STUDY  ANALYSIS  EXAMPLE  1 

Project  1 

Project 

24 

sP 

ov 

in 

IS 

A  | 

Process  Elements  Relevant  to 
Security  TC 

Model 

Data 

Ql.  Match  to  Model? 

Data 

Ql.  Match  to  Model? 

O 

A 

** 

(0 

3 

z 

w 

■E 

O 

*4 

ID 

z 

£ 

oi 

O 

36.  FIPS  199  Security  Categorization 

S 

S 

Y 

D 

N 

89 

53.  Security  Requirements 

Traceability  Matrix 

S 

D 

N 

D 

N 

74 

54.  Plan  of  Actions  and  Milestones 

s 

D 

N 

D 

Y 

85 

55.  System  Security  Plan 

s 

S 

Y 

S 

Y 

100 

56.  Disaster  Recovery  Plan 

s 

S 

Y 

D 

N 

89 

57.  Security  Risk  Assessment  (SRA) 

s 

D 

N 

D 

N 

74 

59.  Security  Test  and  Evaluation  Plan 

s 

D 

N 

D 

Y 

85 

61.  Contingency  Plan 

s 

S 

Y 

S 

Y 

100 

64.  Interconnection  Security 
Agreement 

s 

D 

N 

D 

N 

74 

84.  Security  Assessment  Report 

s 

D 

N 

D 

Y 

85 

85.  Security  Accreditation  Package 

s 

S 

Y 

S 

Y 

100 

92.  Authority  to  Operate  Letter 

s 

S 

Y 

S 

Y 

100 

100.  FISMA  Metrics  Report 

s 

S 

Y 

D 

N 

89 

101.  Security  Incident  Reports 

s 

D 

N 

D 

N 

74 

102.  C&A  Updates 

s 

D 

N 

D 

Y 

85 

Q3.  %  Meeting  Threshold  (must  be  >  75%) 

95% 

Note.  C&A  =  Certification  and  Accreditation;  FIPS  =  Federal  Information  Processing 
Standards;  FISMA  =  Federal  Information  Security  Management  Act. 


Case  Study  Results 

Before  presenting  the  specific  results  of  the  case  study  analysis,  the 
following  key  points  must  be  noted: 
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•  As  stated  in  the  previous  section,  the  analysis  focused  on  the 
project  attributes  that  are  not  the  default.  As  such,  the  table  of 
results  highlights  the  nondefault  attributes  and  the  associated 
comparative  results  to  the  model. 

•  Additionally,  the  Life-cycle  Approach  TC  consisted  of  three 
options:  waterfall,  incremental,  and  agile.  These  three  were 
chosen  based  on  the  common  development  methodologies 
used  in  DHS.  With  waterfall  applied  as  the  default  attribute 
for  the  standard  process,  the  analysis  focused  on  incremental 
and  agile. 

•  In  the  analysis  of  the  11  projects  using  the  incremental  devel¬ 
opment  methodology,  we  discovered  the  tailoring  plan  for  each 
was  limited  to  a  single  increment  of  the  project,  with  a  corre¬ 
sponding  note  that  the  tailoring  plan  would  be  updated  prior 
to  initiation  of  each  new  increment.  As  such,  the  incremental 
development  projects  actually  reflected  a  waterfall-based  tai¬ 
loring  strategy. 

•  For  the  development  methodology,  the  table  of  results  then 
focuses  on  the  projects  that  use  the  agile  method,  and  assesses 
whether  the  model  matches  the  tailoring  approach  of  those 
projects  for  the  process  elements  that  correspond  to  the  Life- 
cycle  Approach  TC. 

Table  6  shows  the  results  of  the  analysis  comparing  the  relevant  projects  to 
the  model  outputs.  The  number  of  projects  with  each  particular  attribute 
is  provided.  Overall,  the  results  show  that  nine  of  the  10  TCs  can  be  used 
to  reduce  the  amount  of  manual  tailoring  for  a  particular  project.  The  one 
exception  is  the  “Project  Size”  TC.  As  discussed  previously,  there  is  limited 
specific  research  regarding  the  application  of  SE  based  on  project  size,  and 
more  specifically,  to  small  projects.  Laporte,  Alexandre,  and  Renault  (2008, 
p.  98)  discuss  the  need  for  developing  international  standards  to  address 
the  needs  of  small  development  organizations;  one  objective  is  to  “provide 
harmonized  documentation”  integrating  standards,  work  products,  and 
deliverables.  This  naturally  extends  to  SE  within  small  organizations  and 
large  organizations  with  small  projects  in  their  portfolio.  Our  analysis 
showed  that  low  correlation  exists  between  the  model  and  the  small  project 
case  studies.  This  occurred  because  of  the  following:  while  very  few  proj¬ 
ects  tailored  the  process  elements  recommended  by  the  model,  each  project 
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team  executed,  on  average,  90  percent  of  the  relevant  process  elements.  As 
a  result  of  their  small  size,  few  projects  leveraged  the  opportunity  to  tailor 
process  elements. 


TABLE  6.  CASE  STUDY  ANALYSIS  RESULTS  ij 

0  Process 

%  Process 

Elements 

Elements 

ID 

Project  Attribute 

#  Relevant 
Projects 

#  Process 
Elements 

With 

>75% 

With 

>75% 

Match  to 

Match  to 

the  Model 

the  Model 

1 

Security  Impact 

17 

15 

15 

100% 

2 

Privacy  Impact 

9 

4 

4 

100% 

3 

Intelligence 

Impact 

3 

1 

1 

100% 

4 

Interoperability 

Impact 

11 

8 

6 

75% 

5 

Accessibility 

Impact 

13 

2 

2 

100% 

Technology 

6 

Demonstration 

7 

1 

1 

100% 

Planned 

7 

Environmental 

Impact 

7 

1 

1 

100% 

8 

Agile 

Development 

2 

19 

15 

79% 

9 

COTS 

6 

9 

7 

78% 

10 

Small  Project 

6 

20 

2 

10% 

Conclusions  and  Future  Work 

Process  tailoring  is  critical  to  effectively  and  efficiently  executing  SE 
based  on  the  unique  characteristics  of  a  project.  Many  SE  processes  recom¬ 
mend  tailoring,  but  are  not  supported  with  tailoring  guidelines.  This  article 
examines  the  previous  research  of  process  tailoring  in  software  literature 
and  SE  standards  for  certain  types  of  projects,  and  proposes  a  model  for 
SE  process  tailoring.  Our  comparison  of  the  SEPTM  model  to  several  case 
studies  within  the  DHS  shows  rule-based  process  tailoring,  coupled  with 
project  attributes,  can  be  a  viable  approach  to  reducing  the  amount  of  man¬ 
ual  tailoring  required  for  a  given  project. 
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While  our  research  objectives  were  accomplished  for  the  development  of 
the  initial  model,  additional  research  should  be  performed  in  the  area  of  SE 
process  tailoring.  Key  elements  requiring  further  research  include: 

•  Very  little  research  exists  regarding  SE  process  execution  on 
small  projects  or  in  small  project  teams.  The  INCOSE  Very 
Small  and  Micro  Entities  (VSME)  Working  Group  is  develop¬ 
ing  work  packages  that  recommend  various  levels  of  SE  process 
execution  for  small  projects.  The  SEPTM  described  herein 
could  be  informed  by  the  findings  of  their  research. 

•  The  model’s  “Complexity  and  Precedence  of  the  System”  TC  is 
based  on  a  selection  of  either  development  or  COTS.  The  model 
should  be  further  refined  based  on  an  investigation  of  COTS 
integration,  which  has  become  more  common  in  recent  years. 

•  The  initial  model  is  based  primarily  on  the  INCOSE  Systems 
Engineering  Handbook.  Certainly  any  number  of  standard  SE 
processes  can  be  used  as  a  starting  point  for  organizational 
process  tailoring;  additional  research  should  be  performed  to 
identify  common  tailoring  considerations  across  a  broader  set 
of  SE  standards  and  literature. 

•  Investigate  applications  of  data  mining  to  the  process  of  select¬ 
ing  the  TCs  that  form  the  basis  of  the  model. 
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The  research  described  herein  aims  to  add  to  the  body  of  knowledge  of 
program  management  and  factors  that  lead  to  acquisition  program  termi¬ 
nations  within  the  U.S.  Department  of  Defense  (DoD).  Specifically,  this 
research  surveyed  three  groups— DoD  acquisition  program  managers,  defense 
industry  program  managers,  and  defense  industry  consultants— to  evaluate 
and  analyze  key  program  factors  that  influence  DoD  acquisition  program 
terminations.  This  research  used  relative  importance  weight  calculations 
and  a  chi-squared  distribution  analysis  to  compare  the  differences  between 
DoD  acquisition  program  managers,  defense  industry  program  managers, 
and  defense  industry  consultants  regarding  the  factors  that  lead  to  DoD 
acquisition  program  terminations.  The  results  of  this  research  indicate  that 
a  statistically  significant  difference  does  not  exist  between  the  three  groups 
as  to  the  relative  importance  of  11  program  management  factors. 


Keywords:  program  management  factors,  relative  importance  weight  calculations,  DoD 
program  termination 
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The  U.S.  Department  of  Defense  (DoD)  loses  billions  of  dollars  annually 
on  canceled  or  failed  acquisition  programs  (DoD,  2013).  In  fact,  many  acqui¬ 
sition  studies  conducted  by  the  Government  Accountability  Office  (GAO), 
DoD,  Office  of  Management  and  Budget,  as  well  as  many  Federally  Funded 
Research  and  Development  Centers  illuminate  the  myriad  of  programs  that 
are  terminated  without  meeting  full  operational  capability  (DoD,  2013). 

From  1997  to  the  present,  DoD  spent  in  excess  of  $62  billion  on  programs 
that  were  eventually  canceled  (Table  1  and  Figure  1).  The  DoD  has  invested 
a  great  deal  of  time,  energy,  and  resources  to  investigate  the  root  causes 
of  program  cancellation  and  to  determine  why  so  many  programs  fail  to 
make  it  through  the  acquisition  system.  In  fact,  The  Office  of  Performance 
Assessments  and  Root  Cause  Analyses  (PARCA),  established  in  2009  by  the 
Weapon  Systems  Acquisition  Reform  Act  of  2009,  continuously  evaluates 
the  status  of  defense  programs  (Weapon  Systems,  2009).  PARCA  issues 
policies,  procedures,  and  guidance  governing  the  conduct  of  such  work  by 
the  Military  Departments  and  Defense  Agencies  (Weapon  Systems,  2009). 
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TABLE  1.  SELECTED  DoD  PROGRAM  TERMINATIONS  AND  COSTS 


Program 

Service 

(Sbillion) 

Source 

Future  Combat  Systems 

Army 

20.00 

GAO 

(2014) 

Joint  Tactical  Radio  System 

Army 

11.00 

Rodriquez 

(2014) 

Comanche  Helicopter 

Army 

5.90 

GAO 

(2014) 

nPOESS  Satellite 

Air  Force 

5.80 

Reed 

(2011) 

Airborne  Laser 

Air  Force 

5.00 

Rodriquez 

(2014) 

VH-71  Presidential  Helicopter 

Marines 

3.30 

GAO 

(2014) 

Expeditionary  Fighting  Vehicle 
(EFV) 

Marines 

3.30 

Reed 

(2011) 

Transformational  SATCOM 

Air  Force 

2.90 

Reed 

(TSAT) 

(2011) 

Crusader 

Army 

2.20 

GAO 

(2014) 

Kinetic  Energy  Interceptor 

Missile  Defense 
Agency 

1.30 

Rodriquez 

(2014) 

Advanced  SEAL  Delivery 

System 

Navy 

0.60 

Reed 

(2011) 

Armed  Reconnaissance 
Helicopter 

Army 

0.50 

Reed 

(2011) 

Aerial  Common  Sensor 

Army 

0.19 

GAO 

(2014) 

CG(X)  Next  Generation  Cruiser 

Navy 

0.20 

Reed 

(2011) 

CSAR-X 

Air  Force 

0.20 

Reed 

(2011) 

TOTAL 

62.39 

Note.  CSAR  =  Combat  Search  and  Rescue;  NPOESS  =  National  Polar-orbiting  Operational 
Environmental  Satellite  System;  SATCOM  =  Satellite  Communications;  SEAL  =  Sea,  Air, 
and  Land. 
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FIGURE  1.  MAJOR  PROGRAMS  OFFICIALLY  CANCELLED  WITHOUT 
PRODUCING  ANY  OR  FEW  OPERATIONAL  UNITS  AS  A  PERCENTAGE 
OF  RDT&E  (1995-2013) 


Note.  (DoD,  2013) 

ACS  =  Aerial  Common  Sensor;  ADS  =  Advanced  Deployable  System;  Alt  =  Alternate;  AMP 
=  Avionics  Modernization  Program;  ARH  =  Armed  Reconnaissance  Helicopter;  ASDS  = 
Advanced  SEAL  Delivery  System;  ATACMS-BAT  =  Army  Tactical  Missile  System  Block  11- 
Brilliant  Anti-Armor;  CSAR  =  Combat  Search  and  Rescue;  DoN  =  Department  of  the  Navy; 
ECSS  =  Expeditionary  Combat  Support  System;  EFV  =  Expeditionary  Fighting  Vehicle; 
ERM  =  Extended  Range  Munition;  FCS  =  Future  Combat  System;  JCM  =  Joint  Common 
Missile;  JTRS  GMR  =  Joint  Tactical  Radio  System  Ground  Mobile  Radio;  MEADS  =  Medium 
Extended  Air  Defense  System;  NECC  =  Navy  Expeditionary  Combat  Command;  NLOS- 
LS  =  Non-Line  of  Sight  Launch  System;  NPOESS  =  National  Polar-orbiting  Observing 
Satellite  System;  SBSS  =  Space  Based  Space  Surveillance;  SLAMRAAM  =  Surfaced- 
Launched  Advanced  Medium  Range  Air-to-Air  Missile;  3GIRS  =  Third  Generation  Infrared 
Surveillance;  TSAT  =  Transformational  Satellite. 


When  programs  are  terminated,  the  DoD  loses  billions  of  investment  dollars 
(Leanard,  2013).  In  some  termination  cases,  the  DoD  garners  value  despite 
termination.  Marginal  benefits  include  economic  value,  knowledge,  skills, 
lessons  learned,  and  insights.  Further,  the  effects  of  termination  influence 
many  areas  of  the  acquisition  enterprise. 
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Scholars,  program  managers,  and  systems  engineers  posit  that  a  host  of 
factors  influences  whether  a  program  is  terminated  or  allowed  to  continue. 
They  include,  but  are  not  limited  to,  political  pressures,  cost  overruns, 
schedule  overruns,  and  performance  shortfalls.  Figure  2  illustrates  the 
various  effects  of  program  termination  (GAO,  2014). 


FIGURE  2.  PROGRAM  CANCELLATION  EFFECTS 


The  purpose  of  this  research  was  to  compare  the  three  groups  that  are 
primarily  associated  with  DoD  program  and  project  management:  DoD 
program  managers,  DoD  program  manager  consultants,  and  DoD  indus¬ 
try  program  managers.  These  three  groups  were  selected  for  comparison 
because  each  works  with  DoD  programs,  but  each  group  has  a  unique  per¬ 
spective.  Exploring  the  different  perspectives  is  essential  for  understanding 
acquisition  systems  (Cornell,  2009).  Viewing  and  understanding  systems 
from  various  perspectives  increase  the  overall  understanding  and  appreci¬ 
ation  of  acquisition  program  system  dynamics  (Cornell,  2009). 
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Most  research  into  DoD  program  termination  focuses  on  an  analysis  of 
scope,  schedule,  and  budget.  This  research  expands  on  those  three  primary 
factors  and  evaluates  eight  other  factors.  Although  other  factors  were  iden¬ 
tified  in  the  literature  review  that  were  not  evaluated,  the  authors  chose  not 
to  evaluate  every  factor  in  the  literature.  Instead,  the  authors  chose  to  eval¬ 
uate  the  11  most  common  factors  across  the  literature  that  explore  program 
success,  failure,  and  termination.  This  research  aims  to  identify  the  factors 
that  have  the  greatest  influence  on  program  and  project  cancellation  from 
the  expert’s  perspective,  and  capture  any  significant  differences  between 
DoD  program  managers,  defense  industry  program  managers,  and  defense 
industry  consultants. 

This  research  aims  to  answer  two  interrelated  research  questions  to  iden¬ 
tify  the  factors  that  have  the  greatest  influence  on  program  and  project 
cancellation  from  the  expert’s  perspective.  The  research  questions  for  this 
article  include: 


1.  Are  there  any  statistically  significant  differences  between 
DoD  acquisition  program  managers,  defense  industry  program 
managers,  and  defense  industry  consultants  as  to  the  leading 
factors  that  result  in  DoD  acquisition  program  terminations? 


2.  What  are  the  critical  factors  and  attributes  that  lead  to  DoD 
acquisition  program  terminations? 

If  statistically  significant  differences  exist  between  the  three 
groups  as  to  the  relative  importance  of  cancellation  factors,  the 
research  will  identify  where  those  differences  exist.  If  statis¬ 
tically  significant  differences  do  not  exist  between  the  three 
groups  as  to  the  relative  importance  of  cancellation  factors, 
the  research  will  identify  the  synergies  between  DoD  pro¬ 
gram  managers  and  defense  industry  program  managers.  In 
the  first  case,  the  differences  could  suggest  future  research 
to  understand  why  different  perspectives  are  prevalent.  In 
the  second  case,  the  common  responses  could  highlight/ 
identify  opportunities  for  emphasis  to  quell  the  frequency 
of  program  terminations. 

Program  and  project  failure  and  success  are  an  enduring  sub¬ 
ject  of  investigation,  discovery,  and  discussion  in  government, 
business,  industry,  and  the  private  sector.  Indeed,  project  ter¬ 
mination  usually  comes  with  tremendous  financial  consequences 
and  significant  loss  of  time.  Within  the  DoD,  a  great  deal  of  research  has 
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been  conducted  by  Federally  Funded  Research  and  Development  Centers 
(RAND,  Center  for  Naval  Analysis,  MITRE,  etc.),  think  tanks,  and  academia 
into  some  of  the  causes  for  program  and  project  failure  (Hofbauer,  Sanders, 
Ellman,  &  Morrow,  2011).  While  most  of  this  research  focuses  on  the  unique 
root  causes  for  individual  program  failure,  a  comprehensive  analysis  at  the 
aggregate  level— using  expert  judgment  to  compare  and  contrast  DoD  pro¬ 
gram  managers,  DoD  industry  program  managers,  and  DoD  consultants— is 
missing  in  the  literature.  This  research  is  the  first  step  in  a  qualitative  and 
quantitative  analysis,  using  expert  judgement  at  the  aggregate  level  through 
a  survey  in  order  to  assess  the  relative  importance  of  recognized  factors  that 
lead  to  program  and  project  termination. 


Literature  Review 


A  key  aspect  of  understanding  program  and  project  failure  is  an  anal¬ 
ysis  of  program  and  project  attributes  and  factors  that  affect  program  and 
project  management.  The  factors  that  influence  program  and  project  man¬ 
agement  success  in  multiple  industries  have  been  thoroughly  investigated 
in  academia.  These  factors  serve  as  an  outstanding  analytical  tool  to  pro¬ 
vide  a  unique  look  into  DoD  acquisition  program  and  project  failures  from 
a  program  and  project  management  perspective.  Essential  to  this  task  is 
identifying  the  key  factors,  understanding  the  root  causes,  and  ascertain¬ 
ing  the  major  influences  of  program  and  project  failure  to  provide  keen 
insight  into  DoD  program  and  project  failure.  Because  DoD  program 
and  project  terminations  cost  American  taxpayers  billions  of 
dollars,  an  investigation  into  this  subject  matter  is  imperative 
for  DoD,  defense  industry,  Congress,  and  systems  engineering 
researchers  in  order  to  glean  an  enhanced  understanding  of 
DoD  program  and  project  failure,  thereby  ensuring  efficient, 
effective,  and  successful  program  and  project  management. 


An  exhaustive  literature  review  identified  11  critical  factors 
associated  with  program  and  project  management  for  exam¬ 
ination.  Program  and  project  management,  project  failure, 
project  success,  and  the  factors  that  lead  to  project  failure 
and  project  success  remain  important  issues  of  significant 
interest  to  program  managers,  decision  makers,  and  executives 
within  the  DoD.  The  literature  is  replete  with  scholarly  articles 
and  research  into  this  endeavor.  The  articles  pertaining  to  factors 
that  impact  project  management,  success,  and  failure  generally  fit  into 
several  broad  categories.  Such  categories  may  include  value  of  project 
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management;  project  success  criteria;  project  failure  criteria;  project  man¬ 
agement  rubrics;  case  studies;  and  industry-specific  research,  consulting 
services,  and  independent  studies,  such  as  information  technology,  con¬ 
struction,  and  engineering.  Further,  a  significant  body  of  research  focuses 
on  the  roles  of  managers  in  project  failure.  The  following  discussion  is  a 
brief  summary  of  the  most  salient  research  on  program  and  project  man¬ 
agement,  failure,  and  success  within  the  literature.  The  11  factors  identified 
in  the  literature  review  served  as  the  factors  for  analysis. 


A  key  aspect  of  understanding  program  and  project 
failure  is  an  analysis  of  program  and  project  attri¬ 
butes  and  factors  that  affect  program  and  project 
management. 

Pinto  and  Slevin  (1987)  developed  a  framework  (Figure  3)  for  understanding 
the  implementation  of  projects,  as  well  as  a  diagnostic  tool  for  the  project 
manager  known  as  the  Project  Implementation  Profile  (Pinto  &  Slevin, 
1987).  Their  research  focused  on  identifying  predictive  factors  of  success¬ 
ful  program  and  project  management,  and  serves  as  a  seminal  work  for  all 
discussions  on  program  and  project  management;  their  research  identified 
the  following  10  factors  (Pinto  &  Slevin,  1987): 

1.  Project  mission 

2.  Top  management  support 

3.  Project  schedule  plan 

4.  Client  consultation 

5.  Personnel  and  recruitment 

6.  Technical  tasks 

7.  Client  acceptance 

8.  Monitoring  and  feedback 

9.  Communication 

10.  Troubleshooting 
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FIGURE  3.  PROJECT  IMPLEMENTATION  PROFILE 


Note.  (Pinto  &  Slevin,  1987) 


Their  framework  showed  that  the  factors  are  dynamic.  Pinto  and  Slevin 
claim  that  when  studying  program  and  project  management,  the  factors 
follow  a  logical  progression.  Despite  recognizing  the  interdependence  of  the 
factors  on  each  other,  their  study  did  not  explore  this  finding. 

Pinto  and  Slevin  further  suggest  that  their  framework  is  an  effective  tool 
for  project  managers.  Project  managers  can  use  their  framework  as  a  means 
to  manage  and  monitor  the  project’s  posture  as  well  as  determining  where 
the  project  is  related  to  its  life  cycle.  Their  tool  can  also  be  used  as  a  mea¬ 
sure  of  project  success.  They  developed  a  Likert  scale  instrument  whereby 
a  project  manager  can  measure  the  importance  of  each  factor  on  a  given 
program  or  project  at  different  points  in  the  life  cycle  to  determine  which 
factor  is  most  important. 

Additional  research  conducted  by  Lawrence  and  Scanlan  (2007)  provides 
tremendous  value  into  project  failure  in  defense  industries.  They  were 
involved  in  a  10-year  research  project  of  U.S.  and  European  aerospace  indus¬ 
tries  to  create  methodologies  and  tools  for  large  aerospace  project  managers 
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(Lawrence  &  Scanlan,  2007).  The  study  was  commissioned  to  address 
a  large  amount  of  project  terminations  in  U.S.  and  European  aerospace 
industries.  Although  their  focus  was  on  aerospace  industries,  the  authors 
maintain  that  their  findings  are  universal  to  large  engineering  projects 
within  all  industries  (Lawrence  &  Scanlan,  2007).  Many  interviews  with 
program  and  project  managers  revealed  that  causes  of  project  termination 
were  not  singularly  the  project  managers’  fault.  Program  and  project  man¬ 
agers  were  characterized  as  highly  intelligent  and  extremely  competent. 
They  concluded  that  more  robust  software  tools  are  needed  to  manage  the 
complexities  of  today’s  multifaceted  engineering  projects.  However,  they 
also  identified  eight  other  critical  elements  that  strongly  impact  project 
success  or  failure  (Lawrence  &  Scanlan,  2007).  They  include  the  following: 

1.  Poor  initial  planning 

2.  Lack  of  clear  objectives  and  deliverables 

3.  Lack  of  understanding  of  dependencies 

4.  Inadequate  resource  allocation 

5.  Poor  risk  analysis 

6.  Poor  change  management 

7.  Lack  of ‘buy- in’  from  stakeholders 

8.  Poor  understanding  of  priorities 

Their  findings  are  germane  to  any  discussion  on  defense  industry  project 
management.  The  technology,  complexity,  large  budgets,  and  multiple  stake¬ 
holders  in  the  aerospace  defense  industry  projects  mirror  the  problems  and 
challenges  oftheDoD  aerospace  acquisition  programs.  Thus,  Lawrence  and 
Scanlan’s  posits  serve  as  a  great  foundation  for  discussion  of  project  termi¬ 
nations  within  the  DoD. 


The  technology,  complexity,  large  budgets,  and  mul¬ 
tiple  stakeholders  in  the  aerospace  defense  industry 
projects  mirror  the  problems  and  challenges  of  the 
DoD  aerospace  acquisition  programs. 
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Research  into  project  management  conducted  by  Mir  and  Pinnington  (2014) 
illustrates  the  dynamic  relationships  and  interactions  of  successful  project 

management  factors.  They  test  the  relationship  _ _ __ 

between  project  management  performance 
and  project  success.  They  concluded  that  a 
positive  correlation  exists  between  project 
management  performance  and  contribut¬ 
ing  variables  of  project  success.  The  project 
management  performance  variables  (Mir  & 

Pinnington,  2014)  included: 

1.  Project  efficiency 

2.  Impact  on  customer 

3.  Impact  on  project  team 

4.  Business  success 

5.  Preparing  for  future 
Project  success  factors  included: 

1.  Project  manager  leadership 

2.  Project  manager  staff 

3.  Project  manager  policy  and  strategy 

4.  Project  manager  partnerships  and  resources 

5.  Project  manager  life  cycle  management  processes 

6.  Project  manager  key  performance  indicators 

Their  research  clearly  showed  that  dynamic  relationships  exist  between 
the  factors.  When  considering  project  management  factors,  a  context  of 
dynamic  relationships  must  be  considered.  Factors  are  not  static;  each 
factor  or  variable  in  a  project  dynamically  influences  other  factors. 

Researched  conducted  by  Allen,  Alleyne,  Farmer,  McRae,  and  Turner  (2014) 
on  project  success  highlights  some  of  the  factors  and  issues  surrounding 
program  success  and  failure.  Using  case  study  analysis  as  the  rubric  to 
identify  project  success  factors,  they  studied  the  U.S.  Coast  Guard’s  123- 
Foot  Patrol  Boat  and  Proctor  and  Gamble’s  New  Growth  factory  (Allen  et 
al.,  2014).  The  researchers  also  developed  a  survey  and  administered  the 
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survey  to  project  managers  involved  in  the  respective  projects.  Based  on  the 
case  studies  and  the  associated  survey,  they  concluded  that  the  following 
factors  influence  project  success  (Allen  et  al.,  2014): 

1.  Project  management  plan 

2.  Responsibility  assignment  matrix 

3.  Budget  monitoring 

4.  Schedule  monitoring 

5.  Insufficient  stakeholder  engagement 

6.  Broad  scope  and  requirements 

7.  Product  monitoring 

They  also  concluded  that  these  factors  are  excellent  tools  for  analysis  on 
large  and  small  projects  (Allen  et  al.,  2014). 

The  Defense  Acquisition  University  Smart  Shutdown  Guidebook  (DAU,  2009) 
provides  tremendous  insights  into  factors  that  lead  to  program  success  or 
failure  that  eventually  lead  to  termination.  The  guidebook  specifically  lists 
the  following  factors: 

1.  Changes  in  threat  environment 

2.  Technology  changes 

3.  Changes  in  budget  environment 

4.  Unsustainable  cost  growth  in  development,  production,  or 
deployment 

5.  Failure  to  meet  key  performance  parameters 

6.  Policy  changes  that  affect  system  deployment 

7.  Selection  of  alternative  approaches  to  mission  requirements 

8.  Shifting  executive  authority  from  one  Service  to  another 
Service 

9.  Other  programmatic  factors 
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These  factors  along  with  other  factors  identified  in  the  literature  review 
serve  as  a  good  basis  for  analysis  of  the  most  influential  factors  for  pro¬ 
gram  termination. 


Although  the  literature  identified  other  factors  that  affect  program  success, 
failure,  and  termination,  the  authors  chose  to  limit  the  scope  of  analysis  of 
this  research  to  the  factors  that  were  most  common  in  multiple  works  of 
the  literature  review.  Table  2  summarizes  the  findings  and  conclusions  of 
these  and  other  researchers  on  the  topic  of  factors  influencing  the  outcomes 
of  acquisition  programs. 
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1  TABLE  2.  11  LEADING  FACTORS  THAT  INFLUENCE  PROJECT  FAILURE  I 

Attributes/Factors 

Sources 

Factor  1:  Schedule-Related  Attributes 

plan  schedule  management, 
defining  activities  and  establishing 
milestones,  sequencing  activities, 
low-speed  decision  making, 
unrealistic  duration,  delays  in 
work  approval,  consistent  and 
compressed  schedule  pressure, 
inability  to  consider  ramp-up  time 

de  Wit  (1988);  Doloi  (2013);  Fogarty 
(2010);  Fox  &  Miller  (2006); 

Frimpong,  Oluwoye,  &  Crawford 
(2003);  Mulcahy  (1999);  Project 
Management  Institute  (2008) 

Factor  2:  Budget-Related  Activities 

cost  management  plan,  budget/cost 
estimation,  budget  determination, 
controlling  costs,  size  of  budget, 
estimating  activity  resources, 
managing  cash  flow,  contractor 
financial  difficulties 

de  Wit  (1988);  Doloi  (2013);  Fox 
&  Miller  (2006);  Frimpong  et  al. 

(2003);  Lawrence  &  Scanlan  (2007); 
Pinto  &  Prescott  (1988);  Pinto  &  Man¬ 
tel  (1990);  Pinto  &  Slevin  (1987);  Proj¬ 
ect  Management  Institute  (2008) 

Factor  3:  Scope/Requirements-Related  Attributes 

vagueness  in  scope,  plan  scope 
management,  requirements  man¬ 
agement  plan,  requirements  collec¬ 
tion,  defining  scope,  well-defined 
work  breakdown  structure,  cli¬ 
ent-initiated  requirements  changes, 
inadequate  scope/requirements 
definition  process,  failure  to  curtail 
scope/requirements  creep,  lack  of 
understanding  the  significance  of 
operational  environment 

Clarke  (1999);  de  Wit  (1988);  Doloi 
(2013);  Fogarty  (2010);  Fox  &  Miller 
(2006);  Frimpong  et  al.  (2003); 
International  Project  Leadership 
Academy  (2016);  Kappelman, 
McKeeman,  &  Zhang  (2006);  Law¬ 
rence  &  Scanlan  (2007);  Mulcahy 
(1999);  Pinto  &  Mantel  (1990);  Pinto 
&  Prescott  (1988);  Pinto  &  Slevin 
(1987);  Project  Management  Insti¬ 
tute  (2008);  Sage  &  Rouse  (2014) 

Factor  4:  Project  Management  Team-Related 

capability  of  firms,  capability  of 

DoD  team,  anticipation  of  design 
changes,  delays  in  receiving 
instructions,  positive  attitudes  of 
participants 

Belassi  &  Tukel  (1996);  Chan  & 
Kumaraswamy  (1997);  Doloi  (2013); 
Fogarty  (2010);  Fox  &  Miller  (2006); 
Frimpong  et  al.  (2003);  Flicks 
(1992);  Kerzner  (1987);  Mansfield, 
Ugwu,  &  Doran  (1994);  Pinto  & 

Mantel  (1990);  Project  Management 
Institute  (2008) 

Factor  5:  Contract-Related 

type  of  contract,  inaccurate 
estimates  in  contract,  form  of 
procurement  and  contractual 
agreements,  poor  contract 
management,  contract  negotiation 

de  Wit  (1988);  Doloi  (2013); 

Frimpong  et  al.  (2003);  Project 
Management  Institute  (2008); 

Shehu  &  Akintoye  (2010) 
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TABLE  2,  CONTINUED  | 

Attributes/Factors 

Sources 

Factor  6:  Planning  Attributes 

not  developing  a  thorough  plan, 
lack  of  planning  buy-in  by  all, 
informal  plan  for  change  requests, 
underestimating  complexity  of 
project,  planning  deficiencies, 
coordinating  ability,  rapport 
between  participants,  selection  of 
program  managers 

de  Wit  (1988);  Fox  &  Miller  (2006); 
International  Project  Leadership 
Academy  (2016);  Kappelman, 
McKeeman,  &  Zhang  (2006); 

Kerzner  (1987);  Lawrence  &  Scanlan 
(2007) 

Factor  7:  Stakeholder  Engagement 

identifying  key  stakeholders, 
stakeholder  management 
plan,  controlling  stakeholder 
engagement,  considering  project 
from  stakeholder  perspective,  failure 
to  get  stakeholder  buy-in  on  major 
decisions,  lack  of  communication 
between  stakeholders 

Fogarty  (2010);  International 

Project  Leadership  Academy  (2016); 
Kerzner  (1987);  Pinto  &  Mantel 
(1990);  Pinto  &  Slevin  (1987);  Project 
Management  Institute  (2008);  Sage 
&  Rouse (2014) 

Factor  8:  Risk  Mitigation 

risk  management,  performing 
qualitative  risk  assessment, 
performing  quantitative  risk 
assessment,  planning  risk 
responses,  controlling  risks,  inability 
to  anticipate  problems 

Clarke  (1999);  Doloi  (2013);  Fox 
&  Miller  (2006);  Frimpong  et 
al.  (2003);  International  Project 
Leadership  Academy  (2016); 

Mulcahy  (1999);  Pinto  &  Prescott 
(1988);  Project  Management 

Institute  (2008) 

Factor  9:  Communication-Related 

communication  between  project 
management  team  members  and 
communication  to  stakeholders 

International  Project  Leadership 
Academy  (2016);  Project 

Management  Institute  (2008) 

Factor  10:  Technology  Readiness  Level  (TRL) 

TRL  level,  shortages  of  technical 
personnel,  delays  in  testing 

Mankins  (2009);  Straub  (2015) 

Factor  11:  Contractor-Related 

inadequate  contractor  experience, 
lack  of  communication  between 
contractor  and  DoD,  subcontractor 
projects,  low  labor  productivity, 
poor  procurement  programming 

Doloi  (2013);  Frimpong  et  al. 

(2003);  Project  Management 

Institute  (2008) 
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Research  Aim  and  Objectives 

This  article  analyzes  and  evaluates  the  causes  of  acquisition  program 
and  project  failure  within  the  DoD.  The  objectives  of  this  article  are: 

•  To  study,  identify,  and  evaluate  the  most  critical  factors  that 
influence  program  and  project  termination  within  DoD; 

•  To  evaluate  the  main  factors,  based  on  expert  judgment,  that 
lead  to  program  and  project  failure,  and  the  relative  importance 
of  those  factors; 

•  To  identify  any  differences  between  DoD  acquisition  program 
managers,  DoD  contractors,  and  DoD  consultants;  and 

•  To  serve  as  a  springboard  for  future  research  in  DoD  program 
and  project  management. 

The  purpose  of  this  research  is  to  expand  the  current  understanding  of 
program  and  project  failures  and  successes,  and  to  identify  the  different 
perspectives  between  various  stakeholders  within  the  acquisition  program 
and  project  management  enterprise  at  the  aggregate  level.  Although  sig¬ 
nificant  research  has  been  conducted  on  terminated  programs  within  the 
DoD,  the  research  has  focused  on  individual  programs  or  a  group  of  select 
programs.  The  Federally  Funded  Research  and  Development  Centers,  GAO, 
and  Congressional  Research  Service  normally  evaluate  a  specific  program 
or  a  small  group  of  programs. 

However,  the  authors  could  find  neither  a  robust  comprehensive  study  based 
on  expert  judgment  (the  approach  used  in  this  research)  in  the  literature, 
nor  the  analytical  approach  used  in  the  text  for  analysis  of  DoD  program 
and  project  terminations. 


Methodology 

For  this  study,  the  examination  and  methodology  used  a  literature 
reviewto  identify  factors  that  lead  to  program  and  project  success  or  failure, 
expert  judgment,  survey,  relative  importance  weight,  and  Chi-squared  dis¬ 
tribution  to  analyze  the  factors  identified  in  the  literature  review.  Relative 
Importance  Weight  (RIW)  methodology  consisted  of  conducting  a  survey 
to  identify  and  evaluate  the  relative  importance  of  the  significant  factors 
influencing  program  termination  (see  Figure  4  for  methodology  flow). 
Respondents  of  this  survey  included  the  following  three  groups:  (a)  DoD 
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program  and  project  managers,  (b)  DoD  industry  personnel,  and  (c)  DoD 
consultants.  If  respondents  did  not  fall  into  one  of  these  groups  or  had  no 
experience  with  program  and  project  termination,  their  responses  were 
not  considered.  The  131  participants  of  a  structured  survey  were  identified 
through  professional  networks,  Project  Management  Institute  events,  and 
National  Defense  Industrial  Association  events. 


FIGURE  4.  METHODOLOGY  FLOW  DIAGRAM 


To  gather  data  for  evaluation,  analysis,  and  comparison  of  program  and 
project  failure  factors  within  the  DoD  program  portfolio,  a  questionnaire 
was  developed  seeking  respondents  from  three  specific  groups:  program 
managers  from  the  Services,  program  managers  from  companies  with  past 
experience  working  on  DoD  programs,  and  DoD  program  managers.  The 
questionnaire  consisted  of  11  leading  factors  that  influence  project  failure, 
extrapolated  from  an  extensive  literature  review.  The  factors  evaluated 
are  outlined  in  Table  2.  The  literature  review  indicated  that  commonality 
existed  between  project  success  and  failure  factors.  The  success  or  failure 
factor  depended  on  the  author’s  point  of  view.  Essentially,  program  success 
and  failure  factors  are  two  sides  of  the  Janus  coin.  In  the  context  of  this 
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text,  program  and  project  failure  is  defined  by  program  termination.  The 
factors  identified  in  the  literature  influence  program  performance  and  thus 
influence  program  termination. 

A  total  of  131  responses  was  analyzed,  which  consisted  of  45  DoD  program 
managers,  52  defense  industry  program  managers,  and  34  defense  industry 
consultants.  Based  on  previous  research  (Doloi  2008;  Flyvbjerg,  Holm,  & 
Buhl,  2004),  these  numbers  are  acceptable  for  this  type  of  analysis.  Further, 
since  these  data  are  ordinal  and  thereby  nonparametric,  many  opinions 
exist  on  what  constitutes  an  appropriate  sample  size  (Bonett  &  Wright, 
2000;  Noether,  1987).  The  various  works  on  estimating  an  appropriate 
sample  size  rely  on  assuming  some  degree  of  normality.  To  be  confident 
in  the  sample  size,  but  maintain  the  integrity  of  the  nonnormality  of  the 
nonparametric  data,  a  sample  size  of  30  was  an  appropriate  sample  for  the 
three  groups.  N=30  is  recognized  in  many  statistical  works  as  an  agreed- 
upon  acceptable  sample  size  (Devore,  2012;  Sprent,  1989).  Table  3  identifies 
the  profiles  of  the  respondents. 
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TABLE  3.  RESPONDENTS’  PROFILES 

Number  of 
Responses 
Used 

Average 

Experience 

(Years) 

Average 
Project  Budget 
(Millions) 

DoD  Program 

Managers 

45 

>10 

>100 

DoD  Defense  Industry 
Program  Managers 

52 

>5 

>100 

DoD  Consultants 

34 

>10 

>100 

Respondents  were  asked  to  rate  the  relative  importance  of  the  factors  that 
influence  project  failure  based  on  a  five-point  Likert  Scale  (1  =  very  low,  2 
=  low,  3  =  medium,  4  =  high,  5  =  very  high).  To  differentiate  the  expert  per¬ 
ceptions  of  the  relative  importance  of  project  failure  between  groups,  two 
hypotheses  were  developed  and  tested: 

•  H  :  There  is  no  agreement  among  groups  of  the  relative  impor¬ 
tance  of  factors  that  influence  program/project  failure. 

•  H  :  Agreement  exists  among  groups  of  the  relative  importance 
of  factors  that  influence  program/project  failure. 


Findings  and  Data  Analysis 

For  analysis  of  responses,  RIW  analysis  was  conducted  (Doloi,  2013; 
Frimpong  et  al.,  2003).  RIW  is  a  weight  measure  to  compare  the  importance 
of  various  attributes  according  to  a  group  of  respondents.  Weights  must  be 
assigned  to  a  collection  of  survey  responses;  if  the  survey  responses  are 
numerical  already,  and  ordered  such  that  the  “most  important”  response  is 
assigned  the  highest  value  (such  as  the  Likert  scale),  the  numerical  assign¬ 
ment  comes  directly  from  the  survey  results.  The  RIW  for  responses  was 
calculated  using  the  following  equation  (Salunkhe  &  Patil,  2013): 


RIW-- 


Xs 


an. 


I"*. 


*100 


(1) 


Relative  Importance  Weight 


RIW.  =  the  relative  weight  important  for  attribute  j 
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a.  =  the  weight  given  to  response  (Likert  is  used,  therefore  i  =  1,2, 3, 4, 5) 
n.  =  the  number  of  people  who  responded  “  i  ”  for  attribute  j . 
x.  =  is  the  sum  of  all  weighted  responses  for  the  jth  attribute. 

N  =  total  number  of  factors 


The  RIW  equation  was  used  to  calculate  the  RIW  for  program  and  project 
failure  factors.  The  weights  were  ranked  for  DoD  program  managers  and 
DoD  contractors.  The  results  of  the  weights  are  shown  in  Table  4. 


TABLE  4.  SUMMARY  OF  RIW  RESPONSES 


Ratings/Rankings 


DoD  Program 
Managers 


DoD  Industry 
Program 
Managers 


DoD 

Consultants 


Factors 

RIW 

Score 

Rank 

RIW 

Score 

Rank 

RIW 

Score 

Rank 

Schedule-Related 

12.86% 

1 

11.71% 

3 

1 

11.66% 

Budget-Related 

10.61% 

2 

12.53% 

1 

2 

11.19% 

Scope-Related 

10.54% 

3 

11.95% 

2 

4 

9.34% 

Project  Management 
Team-Related 

6.15% 

11 

6.15% 

11 

11 

6.94% 

Contract-Related 

6.64% 

10 

6.34% 

10 

5 

8.97% 

Planning-Related 

8.26% 

8 

10.32% 

4 

6 

8.70% 

Stakeholder 

Engagement-Related 

9.47% 

5 

8.26% 

8 

7 

8.33% 

Risk  Mitigation-Related 

9.13% 

7 

8.67% 

7 

9 

7.86% 

Communication-Related 

6.74% 

9 

6.74% 

9 

10 

7.77% 

Technology  Readiness 
Level-Related 

10.43% 

4 

9.13% 

6 

8 

8.14% 

Contractor- Related 

9.16% 

6 

9.16% 

5 

3 

11.10% 

To  determine  if  there  was  a  significant  difference  between  the  rankings  of 
the  three  groups’  responses,  Kendall’s  Coefficient  of  Concordance  served  as 
the  analytical  tool.  Kendall’s  coefficient  of  concordance,  or  Kendall’s  W,  is 
a  nonparametric  statistic,  recognized  as  an  analytical  tool  appropriate  for 
assessing  the  degree  of  agreement  among  judges.  Kendall’s  W  ranges  from 
0  to  1  (Grzegorzewski,  2006).  A  rating  of  zero  indicates  no  agreement  and  a 
rating  of  one  indicates  strong  agreement  (Hollander,  Wolfe,  &  Chicken,  2014): 
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W  = 


12  S 

m 2  (n3  -  ri) 


m  =  total  number  of  judges  (respondents) 
n  =  total  number  of  objects  (factors) 

s=it(RrRy 

2  —  1 


7-1 


Based  on  the  responses  from  Table  4,  Kendall’s  W  =  0.84.  This  strongly 
suggests  that  agreement  exists  among  the  three  groups.  Despite  this  strong 
evidence  of  agreement,  a  Chi-squared  approximation  was  also  conducted 
to  validate  the  results.  The  Chi-squared  equation  is  shown  here,  followed 
by  Table  5. 


X2  =  m(k  -  1) 
(Devore,  2012) 
k  =  number  of  factors 


TABLE  5.  RESULTS  TABLE  ji 

k 

11 

m 

3 

W 

0.84848 

r 

0.77273 

X2 

25.4545 

df 

10 

p-value 

0.00455 

Based  on  the  Chi-squared  equation,  the  calculated  value  of  Chi-squared  was 
25.45.  Using  the  critical  value  for  Chi-squared  for  k  =  11,  degree  of  freedom  = 
10  with  significance  =  .01 ,  the  critical  value  of  yf  was  calculated  as  follows: 
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2(n-l)_  2(10) _  1  O  O 

A05  A05 

Since  25.45  >  18.3,  reject  the  null  hypothesis.  Further,  for  a  level  of  signif¬ 
icance  a  =  .01,  p-value  .0045  is  less  a  =  .01,  reject  the  null  hypothesis.  The 
results  indicate  that  a  significant  level  of  agreement  exists  among  DoD 
program  managers,  DoD  industry  program  managers,  and  DoD  consultants. 


Results  Discussion 

The  survey  was  analyzed  from  the  DoD  program  manager,  DoD  consul¬ 
tant,  and  DoD  industry  program  manager’s  perspective.  RIW  analysis 
illuminated  the  factors  that  have  the  greatest  influence  on  program  termi¬ 
nation  from  the  various  groups’  perspectives.  Table  4  displays  the  rankings 
by  the  various  groups. 


DoD  program  managers  and  defense  industry  pro¬ 
gram  managers  agreed  on  the  top  three  factors  that 
influence  program  termination:  schedule-related 
attributes,  budget-related  attributes,  and  scope-re¬ 
lated  attributes. 

Since  the  data  analysis  indicates  that  agreement  exists  among  the  various 
groups,  the  leading  factors  present  opportunities  to  address.  The  analysis 
indicates  that  several  factors  greatly  influence  program  termination.  DoD 
program  managers  and  defense  industry  program  managers  agreed  on  the 
top  three  factors  that  influence  program  termination:  schedule-related 
attributes,  budget-related  attributes,  and  scope-related  attributes.  DoD 
consultants  ranked  schedule-related  attributes  and  budget-related  attri¬ 
butes  one  and  two  respectively,  but  contractor-related  attributes  was  the 
other  top  three  factor. 

The  data  also  indicate  that  program  and  project  management  team-related 
attributes  was  the  least  most  important  factor  by  all  groups.  This  suggests 
that  strong  agreement  exists  among  the  groups  that  program  and  project 
management  teams  put  forth  great  effort  to  ensure  program  success.  This 
also  infers  that  program  managers  have  the  right  tools  and  understanding 
of  acquisition  systemic  processes  to  be  successful. 
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Recommendations 

Based  on  the  analysis  discussed  previously,  the  authors  offer  several 
recommendations  for  consideration  since  the  experts  agree  that  several 
attributes  influence  DoD  program  termination: 

•  DoD  should  continue  investment  into  understanding  the  root 
causes  of  schedule-related  attributes. 

•  Realistic,  adequate,  and  appropriate  fiduciary  requirements 
must  be  established  early  in  the  programming  process  to 
ensure  program  success. 

•  DoD  should  continue  investment  in  understanding  require¬ 
ments  creep  in  programs. 

•  Since  DoD  consultants  ranked  contractor-related  attributes 
extremely  high,  and  DoD  program  managers  and  DoD  industry 
program  managers  rated  contractor-related  attributes  rela¬ 
tively  high,  this  area  warrants  further  research  to  explore  and 
perform  a  root  cause  analysis  of  contractor-related  attributes. 

•  The  DoD’s  investment  in  program  manager  training  and  equip¬ 
ping  program  managers  should  be  continued. 

Implementation  of  the  recommendations  should  have  a  positive  influence 
on  DoD  acquisition  program  performance. 


Study  Limitations 

The  research  presented  in  this  article  has  two  limitations  that  should 
be  considered  when  digesting  the  findings.  First,  this  study  was  performed 
at  the  aggregate  level  within  the  DoD.  DoD  survey  participants  represented 
all  branches  of  the  Services  and  DoD  program  managers.  Perspectives  from 
the  different  Services  were  not  considered,  but  rather  the  DoD  aggregate. 
Although  the  Services  have  very  similar  experiences  in  program  and  project 
cancellation,  the  nuances  of  the  differences  in  the  importance  of  factors 
is  worth  mentioning  and  exploring  in  future  research.  Another  limitation 
of  the  research  is  the  mode  chosen  for  factor  analysis.  The  researchers 
presented  and  selected  the  factors  for  analysis  to  be  presented  to  survey 
participants.  Although  the  factors  were  determined  from  an  exhaustive 
literature  review,  an  open-ended  survey  may  have  presented  a  new  set  of 
factors  for  analysis  and  consideration  unique  to  DoD  program  and  project 
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management.  Further,  the  researchers  limited  the  factors  for  analysis, 
thereby  excluding  some  factors  from  the  literature.  However,  the  factors 
selected  for  analysis  were  the  factors  most  common  across  multiple  authors 
and  articles. 

Another  limitation  of  the  research  is  further  root  cause  analysis  of  the  fac¬ 
tors  identified,  surveyed,  and  analyzed.  Root  cause  analysis  of  the  factors 
would  provide  greater  fidelity  and  granularity  of  the  factors.  This  fidelity 
and  granularity  could  lead  to  plausible  solutions  and  corrective  actions  to 
address  the  influence  of  these  factors  on  DoD  acquisition  program  termina¬ 
tion.  The  authors  chose  to  first  focus  on  identifying  DoD  acquisition  program 
factors  and  determining  whether  agreement  existed  among  the  three  prom¬ 
inent  DoD  acquisition  groups.  The  authors  recommend  that  future  studies 
should  focus  on  the  root  cause  analysis  of  the  factors  identified. 


Summary  and  Conclusions 

This  research  identified  the  RI W  of  factors  that  influence  DoD  program 
termination.  Factors  were  identified  through  a  literature  review  of  salient 
research  on  factors  that  lead  to  program  success  and  failures.  These  factors 
served  as  the  basis  for  analysis  into  DoD  acquisition  program  termination.  A 
survey  was  developed  from  the  factors  garnered  from  the  literature  review 
to  determine  the  RI  W  of  each  of  the  factors.  The  survey  was  administered  to 
DoD  acquisition  program  managers,  DoD  industry  program  managers,  and 
DoD  consultants.  The  three  groups’  responses  were  compared.  The  results 
showed  that  there  is  agreement  among  the  three  groups  on  the  influence  of 
the  factors  analyzed.  Based  on  the  analysis  of  the  results,  the  authors  pre¬ 
sented  several  recommendations  for  the  DoD  acquisition  enterprise.  This 
agreement  suggests  that  there  are  opportunities  and  areas  for  the  groups  to 
work  together  to  mitigate  the  most  important  factors,  thereby  decreasing 
the  likelihood  of  program  termination. 


Areas  for  Future  Research 

In  a  similar  vein  as  the  study  limitations,  the  authors  recommend  sev¬ 
eral  areas  for  future  research.  First,  this  study  did  not  consider  the  role 
of  the  Congress  in  DoD  acquisition  program  cancellation.  In  the  United 
States,  Congress  plays  a  huge  role  in  program  termination.  Congress  has 
the  power  to  cut  program  budgets,  terminate  programs,  conduct  hearings 
on  program  status,  and  change  requirements.  Often,  the  DoD  wants  to  cut 
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a  program,  but  Congress  orders  the  programs  to  continue.  As  mentioned  in 
the  study  limitations,  an  open-ended  survey  could  produce  an  entirely  new 
set  of  factors  or  attributes  for  consideration  or  analysis  unique  to  DoD  pro¬ 
grams.  Once  these  new  factors  are  identified,  a  host  of  data  analysis  could 
be  performed  including,  but  not  be  limited  to,  dynamic  interactions  of  these 
new  factors,  attribute  and  factor  analysis,  and  RIW.  This  study  identified 
the  most  important  factors.  Future  research  could  focus  on  the  why  of  the 
most  critical  factors  that  are  unique  to  the  DoD.  Another  area  for  future 
research  could  focus  on  the  derivatives  of  failed  and  canceled  programs. 
Although  programs  are  canceled,  a  resultant  loss  is  not  always  incurred.  The 
derivatives,  vestiges,  and  lessons  learned  from  those  programs  suggest  that 
all  is  not  lost.  Putting  a  value  on  these  aspects  could  be  beneficial  in  program 
analysis  or  termination.  For  example,  the  Army  Future  Combat  System  was 
terminated.  On  the  surface  and  aggregate,  this  may  appear  like  a  failure,  but 
many  of  the  technologies  and  systems  developed  were  used  in  other  Army 
systems.  All  was  not  lost  despite  program  failure  and  termination.  A  com¬ 
parison  of  successful  and  failed  DoD  programs  is  another  area  for  future 
research.  This  research  could  compare  the  root  causes  in  the  difference 
between  successes  and  failures.  A  final  area  for  future  research  is  the  role 
of  knowledge  management  in  program  and  project  failure. 
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The  United  States  has  spent  more  than  $23  billion  on  construction  in 
Afghanistan  since  2001.  The  dynamic  security  situation  created  substantial 
project  uncertainty  and  many  construction  projects  used  cost-plus-fixed-fee 
contracts  (CPFF)  instead  of  the  firm-fixed-price  (FFP)  norm.  Using  a  dataset 
of  25  wartime  construction  projects  managed  by  the  Air  Force  Civil  Engineer 
(Renter,  the  authors  sought  to  confirm  that  both  contract  types  yield  project 
outcomes  consistent  with  the  established  literature.  As  expected,  they  found 
CPFF  contracts  had  greater  cost  and  schedule  growth  than  FFP.  However, 
they  did  not  find  differences  regarding  as-built  quality.  Additionally,  the 
authors  sought  to  determine  whether  CPFF  contracts  exhibited  greater 
construction  risks  than  FFP  contracts.  They  found  no  significant  differences 
between  contract  types  in  terms  of  security  incidents  or  other  environmental 
factors.  This  research  maybe  particularly  relevant  to  military  owners  who 
contract  projects  in  wartime  environments. 


Keywords:  firm-fixed-price,  cost-plus-fixed-fee,  Afghanistan,  construction  management, 
contract  structure 


A  Publication  of  the  Defense  Acquisition  University 


http://www.dau.mil 


Background 

Contracts  should  allocate  risk  to  the  contracting  party  best  able  to  man¬ 
age  the  risk.  According  to  Mclnnis  (2001),  risk  in  the  construction  industry 
has  been  categorized  into  two  divisions:  contractual  risk  and  construction 
risk.  Contractual  risks  include  items  such  as  miscommunication,  lack  of 
contract  clarity,  or  poor  contract  administration.  They  are  internal  to  the 
contract  and  occur  because  an  imperfect  owner  and  imperfect  contractor 
have  chosen  to  work  together.  Construction  risk  includes  items  such  as 
weather,  resource  availability,  and  acts  of  God.  In  contrast  to  contractual 
risk,  construction  risk  is  external  to  the  contracting  parties  and  would 
exist  even  if  the  parties  were  perfect  (Mclnnis,  2001).  Risk  allocation  is 
especially  important  in  a  wartime  construction  environment.  Contractors 
working  on  behalf  of  the  U.S.  mission  in  Afghanistan  faced  a  host  of  risks, 
including  security  threats,  long  logistical  chains,  extreme  weather,  and  a 
lack  of  qualified  personnel  (Recurring  Problems  in  Afghan  Construction, 
2011).  As  the  owner,  the  U.S.  Government  managed  and  allocated  the  risk 
through  contract  type  choices.  Using  the  lens  of  contract  types  employed 
in  Afghanistan,  namely  fixed-price  and  cost-reimbursable  contracts,  this 
article  seeks  to  understand  better  the  differences  in  contractor  behaviors 
across  contract  types  in  a  wartime  construction  environment. 


The  default  contract  type  for  federal  construction  services  is  a  firm-fixed- 
price  (FFP)  contract  (Federal  Acquisition  Regulation  [FAR], 

2015,  pt.  36.207).  Contracting  officers  are  responsible  for 
contract  type  determinations,  but  they  are  aided  by  the  FAR 
decision  framework.  Contracts  with  well-established  speci¬ 
fications  that  allow  both  the  government 
and  prospective  bidders  to  estimate 
costs  accurately  should  be  FFP  con¬ 
tracts  (FAR,  2015,  pt.  16.104;  Scherer, 

1964);  such  contracts  place  the  max¬ 
imum  amount  of  construction  risk 
on  the  contractor  and  should  also 
provide  the  contractor  with  higher 
profit  expectations  (Scherer,  1964). 

Construction  contracts  typically  are  rea¬ 
sonably  well  defined.  Thus,  FFP  contracts 
should  be  used. 
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A  cost-plus-fixed-fee  (CPFF)  contract  is  the  opposite  of  an  FFP  contract. 
The  government  assumes  all  risk  for  allowable  costs  up  to  the  extent  pre¬ 
scribed  in  the  contract  (FAR,  2015,  pts.  16.301-1).  Under  this  contract  type, 
the  government  and  contractor  agree  to  a  fee  that  is  fixed  at  the  inception  of 
the  contract  and  based  on  an  estimate  of  total  costs  rather  than  final  costs 
(FAR,  2015,  pt.  16.306).  The  estimate  is  not  binding,  and  the  true  cost  is 
flexible  up  to  the  allowed  maximum  amount  (Scherer,  1964).  Unless  signif¬ 
icant  uncertainty  or  risk  is  involved  in  the  project,  CPFF  contracts  should 
not  be  used  in  federal  procurement  (FAR,  2015,  pts.  16.301-2).  Because  the 
government  bears  the  risk  of  the  uncertain  environment,  these  contracts 
have  significantly  weaker  cost-efficiency  incentives  (Scherer,  1964);  conse¬ 
quently,  they  are  typically  used  only  for  preliminary  and  exploratory  studies 
as  a  precursor  to  FFP  contracts  (FAR,  2015,  pt.  16.306). 


An  incentive  contract  is  a  third  type  of  contract  that  lies  between  the  polar 
opposites  of  FFP  and  CPFF  contracts.  This  contract  type  allows  owners  to 
reward  contractors  for  meeting  specific  cost,  delivery,  or  performance  goals 
(Bower,  Ashby,  Gerald,  &  Smyk,  2002).  According  to  the  FAR  (2015, 
pt.  16.401),  incentive  contracts  may  be  used  when  FFP  con¬ 
tracts  are  not  feasible  and  the  government  needs  options  to 
motivate  the  contractor  to  improve  delivery  efficiency  and 
minimize  waste.  The  contract  type  allows  owners  to  share 
risk  more  evenly  with  contractors.  Incentive  contracts 
have  increased  in  popularity  in  the  private  construction 
sector  (In’t  Veld  &  Peeters,  1989).  Yet,  their  actual  usage 
remains  low  in  absolute  terms.  The  literature  suggests  that 
owners  will  generally  use  either  FFP  or  CPFF  contracts  for 
construction  services  (Bajari  &  Tadelis,  2001). 

From  2002  to  2013,  the  United  States  spent  more  than 
$23  billion  on  wartime  construction  efforts  in 
Afghanistan  (Johnson,  2014,  p.  2).  Contractors  build¬ 
ing  and  repairing  infrastructure  and  facilities  on 
behalf  of  the  U.S.  Government  faced  a  different 
and  unique  environment  when  compared  to 
peacetime  construction.  This  environment 
included  Taliban  attacks  that  killed  or 
injured  workers  and  destroyed  equipment 
(Affleck,  Seman,  Deegan,  Freeman,  & 
Sargand,  2011;  cf.  Tawazuh  Commercial  and 
Construction  Co.  Ltd.  v.  United  States,  2011),  a  remote  and 
problematic  supply  chain  (Boon,  Huq,  Lovelace,  2011;  cf. 
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Water  Reclaim  Systems  Inc.  v.  United  States,  2008),  and  a  harsh  physical 
environment  (Affleck  et  al.,  2011).  The  environment  was  deemed  sufficiently 
uncertain  that  the  U.S.  Government’s  contracting  officers  elected  to  use  a 
combination  of  FFP  and  CPFF  contracts  to  support  U.S.  construction 
requirements. 


While  cost  reimbursable  contracts  may  entice  com¬ 
panies  to  submit  bids,  they  also  provide  a  significant 
possibility  for  cost  growth  and  will  need  to  be  moni¬ 
tored  differently  than  fixed-price  contracts. 


We  seek  to  expand  the  body  of  knowledge  regarding  contract  types  in  war¬ 
time  construction.  This  research  may  be  particularly  relevant  to  military 
owners  who  contract  construction  projects  in  wartime  environments. 
While  cost  reimbursable  contracts  may  entice  companies  to  submit  bids, 
they  also  provide  a  significant  possibility  for  cost  growth  and  will  need  to 
be  monitored  differently  than  fixed-price  contracts.  Conversely,  fixed-price 
contracts  in  wartime  environments  may  shift  so  much  risk  on  contractors 
that  it  is  impossible  for  companies  to  make  a  profit,  leading  to  higher  prices 
due  to  a  lack  of  competitive  bids  or  a  reduction  in  project  quality.  Therefore, 
this  research  effort  used  data  from  25  Afghan  wartime  construction  projects 
to  search  for  factor  differences  between  fixed-price  and  cost-reimbursable 
projects.  These  projects  were  funded  by  the  U.S.  Government  in  support  of 
the  NATO  Training  Mission-Afghanistan,  with  the  Air  Force  Civil  Engineer 
Center  (AFCEC)  serving  as  the  construction  agent  (i.e.,  the  entity  respon¬ 
sible  for  contract  administration,  including  quality  assurance).  While 
the  Afghan  government  took  ownership  after  contract  close  out,  the  U.S. 
Government  was  the  owner  during  contract  administration.  We  seek  to 
answer  two  investigative  questions  in  this  study: 

1.  Do  CPFF  and  FFP  wartime  construction  contracts  yield  proj¬ 
ect  outcomes  consistent  with  the  established  (peacetime) 
literature? 

2.  Given  that  CPFF  contracts  should  be  used  in  uncertain  cir¬ 
cumstances,  did  CPFF  contracts  exhibit  greater  construction 
risks  than  FFP  contracts? 
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Literature  Review 

The  underlying  theory  of  contract  behavior  based  on  contract  types  has 
been  well  established.  The  theory  of  contractual  incentives  promulgated 
by  Sherer  (1964)  established  expected  contractor  behaviors  using  a  max¬ 
imization  problem.  The  theory  focuses  on  expected  contractor  behaviors 
in  incentive  contracts  (cf.  Federal  Acquisition  Regulation,  2015,  pt.  16.401) 
that  lie  between  the  polar  choices  of  FFP  and  CPFF,  yet  it  also  informs  our 
understanding  of  contractor  behaviors  in  FFP  and  CPFF  contracts.  For 
all  contract  types,  a  contractor’s  profit,  n  ,  can  be  determined  by  using  the 
following  equation,  where  nT  equals  the  target  profit  amount,  a  equals  the 
cost-sharing  coefficient,  CT is  the  negotiated  target  cost,  and  CA  is  the  actual 
cost  charged  to  the  contract. 


nc=nT  +  a(CT  -  CA) 

For  FFP  contracts  a  will  equal  one,  and  for  CPFF  contracts  a  will  equal 
zero.  Simplifying  the  equation,  we  see  that  a  contractor’s  expected  profit  for 
FFP  contracts  is  its  negotiated  target  amount  plus  its  bid  price  minus  actual 
costs.  In  contrast,  for  CPFF  contracts,  a  contractor’s  expected  profit  is  only 
the  negotiated  target  amount  (which  may  increase  through  the  negotiation 
of  added  work).  Hence,  it  is  widely  known  that  there  are  weak  cost-saving 
incentives  for  CPFF  contracts. 

The  negotiated  target  amount,  n  is  a  function  of  financial  risk.  When  the 
contractor  bears  additional  financial  risk,  such  as  in  an  FFP  contract,  the 
negotiated  target  amount  will  be  higher.  When  the  contractor  has  negligible 
risk,  as  in  the  case  of  CPFF,  the  target  amount  will  be  lower  (Scherer,  1964). 

Shearer’s  (1964)  study  notes  several  key  contractor  behaviors.  First,  for 
established  projects,  where  the  risk  can  be  managed,  contractors  should 
prefer  FFP  contracts  as  they  have  higher  potential  profit  margins  for  the 
contractor.  Second,  as  project  uncertainty  increases,  contractors  prefer 
CPFF  contracts  over  FFP,  at  the  expense  of  higher  profit  margins;  CPFF 
contracts  shield  contractors  from  potential  losses  due  to  the  uncertainty. 
Last,  because  FFP  contractors  bear  the  risk  for  actual  costs,  C , ,  if  con- 
tractors  encounter  unexpected  risk,  actual  costs  can  be  reduced  by  cutting 
quality,  letting  the  schedule  slip,  or  eliminating  personnel. 

Bajari  and  Tadelis  (2001)  have  proposed  a  complementary  theory  that  views 
the  contract-type  decision  in  terms  of  postaward  adaptability  instead  of 
preaward  superior  knowledge.  They  note  that  FFP  contracts  can  reduce 
initial  costs,  but  those  cost  savings  can  be  lost  through  contract 
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modifications.  Cost  savings  are  lost  because  the  FFP  contract  compensation 
scheme  is  based  on  specific  delivery  requirements  agreed  to  within  the 
contract.  While  the  contract  allows  for  changes,  implementation  of  the 
changes  requires  the  compensation  to  be  renegotiated.  In  contrast,  CPFF 
contracts  have  a  well-defined  compensation  scheme  for  both  the  initial 
design  and  subsequent  changes.  Awarding  a  contract  modification  does  not 
require  renegotiation.  The  postaward  adaptability  also  implies  that  less 
conflict  (or  friction)  will  be  observed  between  the  owner  and  the  contractor 
with  CPFF  contracts.  Uncertainty  also  plays  a  central  theme  in  Bajari  and 
Tadelis’s  model.  For  projects  with  little  or  no  uncertainty,  FFP  contracts 
will  be  preferred;  for  projects  with  high  uncertainty,  CPFF  contracts  will 
be  preferred.  The  model  suggests  that  complex  projects  should  be  acquired 
using  CPFF  contracts  to  allow  greater  adaptability  to  the  inherent  design 
changes;  in  contrast,  simpler  projects  should  be  acquired  using  FFP  con¬ 
tracts  to  provide  cost  savings  to  the  owner. 


The  literature  is  clear  that  as  project  requirement 
uncertainty  increases,  owners  should  consider  the 
use  of  cost-reimbursable  contracts. 


The  empirical  evidence  within  the  literature  largely  supports  these  two 
theories.  FFP  contracts  should  be  used  for  well-defined  projects  and  CPFF 
contracts  for  projects  with  more  uncertainty  (Adler  &  Scherer,  2011;  In ’t 
Veld  &  Peeters,  1989;  Muller  &  Turner,  2005;  von  Branconi  &  Loch,  2004; 
Wamuziri,  2013).  First,  von  Branconi  and  Loch  (2004)  and  Muller  and  Turner 
(2005)  discussed  the  term  project  uncertainty,  i.e.,  the  project’s  degree  of 
risk,  using  the  framework  of  owner  involvement.  Those  authors  observe  that 
owners  tend  to  be  less  involved  during  FFP  construction,  which  can  lead  to 
perceived  poor  outcomes.  Because  the  project  requirement  is  expected  to  be 
well  defined,  an  owner’s  failure  to  apply  sufficient  diligence  in  defining  the 
requirement  may  lead  to  an  outcome  that  does  not  meet  quality  expectations. 
Von  Branconi  and  Loch  (2004)  and  Muller  and  Turner  (2005)  also  note 
that  with  CPFF  contracts,  the  project  is  ill-defined  by  definition.  The  lack 
of  definition  compels  the  owner  to  be  more  involved,  resulting  in  physical 
outcomes  that  typically  meet  expectations.  As  is  often  the  case,  if  costs  are 
not  controlled,  CPFF  will  have  higher  costs.  Adler  and  Scherer  (2011)  view 
the  uncertainty  difference  in  terms  of  knowledge.  If  the  contractor  can  apply 
superior  knowledge  in  support  of  the  contract  requirements,  CPFF  contracts 
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are  preferred;  otherwise,  FFP  con¬ 
tracts  should  be  used.  Lastly,  In ’t  Veld 
and  Peeters  (1989)  examined  which 
categories  of  construction  uncertainty 
should  sway  the  contract  type  decision. 
They  found  that  FFP  contracts  were 
an  appropriate  mechanism  for  con¬ 
tractors  to  manage  risk  from  resource 
availability,  schedule  criticality,  and 
performance  requirements.  However, 
if  the  risk  is  due  to  cost  uncertainty 
or  technical  uncertainty,  cost-reim¬ 
bursable  contracts  should  be  used. 
The  literature  is  clear  that  as  project 
requirement  uncertainty  increases, 
owners  should  consider  the  use  of 
cost-reimbursable  contracts 


Scherer’s  theory  as  it  relates  to  cost 
performance  and  quality  has  largely 
been  substantiated  in  recent  work 
investigating  construction  contract¬ 
ing.  Wamuziri  (2013)  found  that 
negotiated  target  amounts  are  indeed 
higher  for  FFP  construction  projects. 

Additionally,  he  found  CPFF  con¬ 
tracts  to  have  higher  overall  costs. 

Jaszkowiak  (2012)  conducted  the  only 
wartime  comparison  of  contract  types  that  we  were  able  to  locate.  She  found 
that  FFP  contracts  had  less  schedule  growth,  CPFF  contracts  produced 
better  quality  facilities,  and  there  was  no  cost  growth  difference  between 
the  two.  While  the  study  had  a  small  sample  size,  the  results  are  generally 
consistent  with  previous  literature,  with  the  exception  that  she  did  not 
observe  cost  growth  differences. 


In  summary,  the  literature  suggests  three  primary  performance  differences 
between  FFP  and  CPFF  contracts.  First,  on  average,  FFP  contracts  will  have 
less  cost  growth  than  CPFF  contracts.  Second,  FFP  contracts  will  have  less 
schedule  growth  than  CPFF  contracts.  Lastly,  FFP  contracts  will  be  of  lesser 
quality  than  CPFF  contracts.  These  three  factors— time,  cost,  and  quality- 
form  the  project  management  iron  triangle  and  are  known  to  influence  one 
another  (Ika,  2009). 
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Given  these  performance  outcomes  as  indicated 
by  the  literature,  we  next  will  discuss  how  war¬ 
time  construction  may  differ  from  peacetime 
construction  and  suggest  ways  in  which  the  per¬ 
formance  differences  maybe  affected.  Wartime 
projects  likely  face  the  same  risks  as  peacetime 
projects,  with  some  notable  additions.  The 
U.S.  Army  Corps  of  Engineers  (USACE)  com¬ 
missioned  a  study  to  document  construction 
challenges  in  Afghanistan  (Affleck  et  al.,  2011). 

Many  of  the  risks  observed  by  the  USACE  are 
not  unique  to  wartime— they  are  common  in  other 
nearby  Asian  and  African  countries  and  include 
design  problems,  planning  problems,  weather  inter¬ 
ference,  unskilled  workers/quality  problems,  difficulty 
working  with  the  owner  or  lack  of  direction  from  the  owner, 
and  change  orders  or  scope  changes  (Affleck  et  al.,  2011: 
Assaf  &  Al-Hejji,  2006;  Mansfield,  Ugwu,  &  Doran,  1994; 
Marzouk  &  El-Rasas,  2014;  Olima  &  K’akumu,  1999). 


The  Afghanistan  Study  found  that  security  concerns  were 
overwhelmingly  the  primary  challenge  to  projects  (Affleck 
et  al.,  2011).  This  factor  is  unique  to  wartime  projects. 

However,  the  FAR  contains  provisions  for  security. 

It  defines  acts  of  God  (weather)  and  acts  of  the  pub¬ 
lic  enemy  (hostile  or  criminal  acts)  as  excusable  delays  (FAR,  2015,  pts. 
52.249-14;  Kelleher,  Walters,  Smith,  Currie,  &  Hancock,  2009).  Also,  while 
not  required  by  the  FAR,  it  was  common  practice  to  require  contractors  to 
carry  insurance  to  cover  the  loss  of  equipment  stemming  from  criminal  or 
hostile  acts.  Additionally,  many  contracts  required  contractors  to  provide 
their  own  security,  because  U.S.  military  and  Afghan  Security  Forces  did 
not  provide  active  security  for  construction  projects  ( Tawazuh  Commercial 
and  Construction  Co.  Ltd.  v.  United  States,  2011).  In  the  context  of  contract 
types  and  risk  allocation,  the  contracts  treated  the  security  as  a  valid  con¬ 
struction  risk. 


In  assessing  the  resulting  risk,  arguments  can  be  made  for  classifying  a 
project  as  either  an  FFP  or  a  CPFF  contract.  One  argument  for  continuing  to 
classify  construction  as  FFP  is  that  the  project  specifications  do  not  change 
as  a  result  of  possible  attacks.  Technical  uncertainty  would  remain  the  same 
(In ’t  Veld  &  Peeters,  1989).  However,  using  the  cost  uncertainty  argument 
(In ’t  Veld  and  Peeters,  1989),  one  could  argue  that  security  risks  will  cause 
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more  cost  uncertainty.  Even  when  the  company  is  insured  against  the  loss 
of  personnel  or  equipment,  the  cash  needed  to  continue  the  project  could  be 
at  risk  while  the  claim  is  adjudicated.  Without  sufficient  cash  to  continue 
material  acquisition  and  payroll  requirements,  a  project  could  be 
halted  while  it  is  made  whole.  Thus,  it  is  reasonable  to  use  a  CPFF 
contract  to  cope  with  cost  uncertainty. 

The  physical  environment  of  the  project  is  commonly  mentioned 
in  both  wartime  and  peacetime  literature.  Weather  conditions 
are  one  of  the  most  commonly  cited  delay  factors  for  all  projects. 
Afghanistan  has  the  potential  for  particularly  harsh  weather, 
especially  in  the  mountainous  regions.  Affleck  et  al.  (2011) 
stated  that  planning  for  harsh  weather  was  particularly  poor  in 
Afghanistan.  Other  industry  literature  does  not  discuss  planning,  but  does 
consistently  cite  weather  as  a  cause  for  delay.  Most  construction  contracts 
allow  for  a  certain  number  of  weather  delay  days,  but  also  state  that  it 
is  considered  an  excusable  delay,  offering  no  compensation  except  in 
extreme  cases  (Kelleher  et  al.,  2009).  As  the  literature  notes,  sched¬ 
ule  criticality  can  be  effectively  managed  with  FFP  contracts 
(in  ‘t  Veld  &  Peeters,  1989).  Notwithstanding  the  harsh 
environment,  there  is  no  compelling  argument  for  CPFF 
contracts  instead  of  FFP  contracts. 


Methodology 

To  understand  how  contract  types  affect  project  outcomes  (i.e.,  sched¬ 
ule,  cost,  or  quality)  in  wartime  construction  projects,  the  Mann-Whitney 
median  comparison  test  was  used  to  test  differences  among  the  median  for 
project  factors  and  performance  factors  (Table  1).  The  project  factors  are 
basic  metadata  relating  to  cost  and  schedule  performance  for  each  project, 
such  as  award,  contract  length,  and  the  number  of  contract  modifications. 
Performance  factors  relate  to  quality  performance:  the  major  construction, 
design,  and  material  quality  control  deficiencies  cited  by  the  quality  assur¬ 
ance  engineer,  as  well  as  worker  health  and  safety  compliance.  Note  that 
within  Table  1,  the  performance  factors  are  subdivided  by  major  construc¬ 
tion  elements  and  represent  observed  deficiencies  by  government  quality 
assurance  (QA)  engineers.  As  the  FAR  contains  clauses  to  accommodate 
contingency  construction,  we  expect  to  see  project  outcomes  similar  to 
those  described  by  the  literature. 
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TABLE  1.  PERFORMANCE  ANALYSIS  FACTORS 


Project  Factors 

Award  Amount 

Final  Cost 

Awarded  Cost  Growth  (Index) 

Number  of  Contract  Modifications 

Number  of  Change  Orders  (Scope  Changes) 

Number  of  FPOP  Extensions 

Total  Days  Added  to  the  Contract 

Initial  Period  of  Performance 

Final  Period  of  Performance 

Awarded  Schedule  Growth  (Index) 

Performance  Factors 

Quality  Factors 

Horizontal  Work  (concrete  and/or  asphalt) 

Building  Foundation  (concrete/rebar/soils) 

Electrical  (high  and  low  voltage,  comm  lines/outlets) 
Mechanical  (HVAC,  gas,  boilers) 

Utility  (water,  sewer,  and  storm) 

Structural  (masonry,  steel,  and  wood) 

Interior  Finishing  (doors,  tiles,  walls,  ceilings,  bathroom 
fixtures,  paint) 

Exterior  Finishing  (windows,  exterior  doors,  garage 
doors,  fences) 

Technical 

Performance 

Factors 

Design  Performance 

Material/Submittals 

Health  and 

Safety 

Safety  Incidents  and/or  Deficiencies 

The  Mann-Whitney  median  comparison  test  was  used  to  test  differences 
among  the  median  for  uncertain  environmental  factors  to  determine 
whether  FFP  and  CPFF  contracts  exhibited  similar  levels  of  external  con¬ 
struction  risks  (Table  2).  Environmental  factors  are  the  external  elements 
of  the  physical  setting  that  are  outside  the  control  of  the  contractor.  Taliban 
attacks,  severe  weather,  and  interference  from  the  Afghan  government  are 
examples  of  external  environment  factors.  We  expect  that  CPFF  contracts 
should  have  more  instances  of  security  or  weather  challenges  to  account  for 
the  greater  construction  uncertainty. 
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TABLE  2.  RISK  ANALYSIS  FACTORS 


External  Environmental  Factors 

Region  of  Afghanistan 

Security  Incidents 

Other  External  Environment  Issues 

Weather 


The  response  variables  were  obtained  by  analyzing  each  project’s  daily 
reports,  created  by  the  U.S.  Government’s  QA  engineers.  Twenty-five  proj¬ 
ects  were  analyzed:  11  were  FFP,  and  14  were  CPFF.  All  projects  were 
managed  by  AFCEC  and  were  in  support  of  the  Afghan  Government. 
Consequently,  all  projects  were  considered  “outside-the-wire”  (i.e.,  they 
occurred  outside  of  the  guarded  perimeter  of  U.S.  military  operating  loca¬ 
tions).  Each  report  contained  comments  regarding  construction  quality 
(positive  and  negative)  as  well  as  daily  construction  activities  (e.g.,  quality 
deficiencies,  mock-up  meetings,  progress  for  each  craft).  They  also  doc¬ 
umented  delays,  security  incidents,  safety  mishaps,  or  deficiencies.  The 
average  award  cost  was  $25.5  million  (median  was  $17.0  million),  and  the 
average  final  cost  of  the  projects  was  $33.2  million  (median  was  23.9  mil¬ 
lion).  The  majority  of  the  projects  focused  on  vertical  construction.  Table  3 
provides  summary  data  regarding  the  projects. 


TABLE  3.  PROJECT  DATA  j 

Project  Information 

Mean 

Median 

Standard 

Deviation 

Award  Amount 

$25.5  M 

$17.0  M 

$21.4  M 

Final  Cost 

$33.2  M 

$23.9  M 

$28.7  M 

Number  of  Contract  Modifications 

8.73 

7 

3.94 

Change  Orders  (Scope  Changes) 

2.93 

2 

2.40 

Initial  Period  of  Performance  (days) 

382.76 

365 

144.82 

Final  Period  of  Performance  (days) 

822.84 

741 

353.70 

The  daily  quality  reports  were  coded  by  the  factors  shown  in  Tables  1  and  2, 
yielding  the  independent  variables  for  this  study.  When  an  occurrence  of  a 
factor  was  encountered  in  the  review  of  the  daily  reports,  the  incident  was 
recorded.  Each  occurrence  was  independently  linked  to  the  project  and 
all  of  the  metadata  associated  with  that  project.  This  linkage  allowed  for 
a  summary  coding  for  each  project,  which  then  allowed  for  differentiation 
between  projects,  based  on  contract  type. 
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Typically,  two  samples  are  compared  for  differences  using  a  i-test;  it  cal¬ 
culates  a  mean  and  standard  error  based  on  a  significance  level  for  each 
sample.  The  standard  errors  are  then  compared  to  see  whether  the  error 
bands  overlap.  If  they  overlap,  one  can  conclude  that  there  is  no  significant 
difference  between  the  samples.  The  i-test  requires  the  assumption  of 
normally  distributed  data.  Since  our  data  did  not  meet  this  assumption, 
we  used  the  Mann-Whitney  test  (also  known  as  the  Wilcoxon  rank-sum 
test)  to  determine  whether  there  were  performance  differences  between 
the  contract  types.  Conceptually,  the  main  difference  between  a  t-test  and 
the  Mann-Whitney  test  is  the  latter’s  use  of  relative  values  compared  to 
observed  values  in  a  t-test.  In  the  Mann-Whitney  test,  the  observed  values 
are  converted  to  relative  values  by  rank  ordering  them  from  1  to  n.  A  sum- 
rank  score  is  then  calculated  that  is  then  converted  to  a  hypothesis  test 
statistic,  U,  and  used  in  a  standard  .z-test  (Gold,  2007). 


If  a  i-test  is  used  and  its  assumptions  are  violated,  it  can  cause  the  analyst 
to  draw  incorrect  conclusions.  Consider  the  case  in  which  the  data  are  not 
normally  distributed,  but  contain  outliers  to  the  right  (i.e.,  final  period 
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of  performance  with  a  mean  of  823  days,  a  median  of  741,  and  a  standard 
deviation  of 354).  The  increased  variance  from  the  outliers  will  increase  the 
standard  error  and  cause  the  error  band  to  be  larger.  As  error  bands  grow 
larger,  statistical  differences  are  more  difficult  to  observe.  Thus,  one  could 
infer  there  is  no  difference  between  the  samples  when  there  really  is  a  differ¬ 
ence.  With  the  Mann-Whitney  test,  the  influence  of  outliers  is  diminished 
because  each  observation  is  compared  to  other  observations  relatively;  it  is 
a  more  robust  test  than  the  t-test.  When  the  data  are  normally  distributed, 
the  Mann-Whitney  test  has  an  asymptotic  efficiency  of  approximately  95 
percent  when  compared  to  a  t-test  (Lehmann,  2006). 

Thus,  the  Mann-Whitney  hypothesis  test  was  used  to  determine  whether 
the  median  values  for  each  contract  type  were  statistically  different;  its 
solution  can  indicate  whether  there  is  a  significant  difference  in  construc¬ 
tion  outcomes  as  measured  by  the  average  performance  of  an  FFP  contract 
over  a  CPFF  contract.  Its  application  is  appropriate  for  our  data,  which  are 
not  normally  distributed. 


Even  more  than  security,  the  weather  was  the  most 
commonly  reported  external  environment  issue, 
followed  by  security  incidents,  and  then  by  any  other 
external  environmental  issue,  which  ranged  from  lo¬ 
cals  and  the  Afghan  National  Army  interfering  with 
the  project,  to  a  swine  flu  outbreak  halting  progress 
on  several  projects  for  multiple  days. 

Analysis 

All  of  the  projects  exhibited  a  significant  amount  of  construction  risk.  Even 
more  than  security,  the  weather  was  the  most  commonly  reported  external 
environment  issue,  followed  by  security  incidents,  and  then  by  any  other 
external  environmental  issue,  which  ranged  from  locals  and  the  Afghan 
National  Army  interfering  with  the  project,  to  a  swine  flu  outbreak  halting 
progress  on  several  projects  for  multiple  days.  Most  projects  had  fewer 
than  40  days  of  weather  delays.  The  maximum  number  of  delay  days  due 
to  security  was  18.  However,  the  majority  of  the  projects  had  fewer  than  6 
days  cited.  A  summary  is  shown  in  Table  4,  and  an  accompanying  histogram 
appears  in  Figure  1. 
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TABLE  4.  EXTERNAL  ENVIRONMENTAL  FACTORS  j 

Factor 

Mean 

Median 

Standard 

Deviation 

Weather  (days  lost) 

20.32 

12 

21.95 

Security  Incidents  (days  lost) 

5.32 

3 

6.08 

Other  External  Environmental  Issues 
(days  lost) 

6.60 

1 

11.70 

Significant  variance  in  the  number  of  quality  deficiencies  was  also  noted 
among  the  projects.  The  most  common  performance  problems  were  with 
the  design  and  material  submittals  of  a  project.  There  were  no  recorded 
incidents  of  poor  engineering  that  led  to  a  failure.  However,  because  the 
government  had  a  thorough  review  process,  the  most  commonly  observed 
problem  was  contractors  submitting  finalized  designs  that  did  not  address 
all  the  review  comments,  causing  many  unnecessary  revision  and  resub¬ 
mission  cycles.  The  majority  of  projects  had  between  0  and  15  design 
performance  incidents,  and  one  project  had  31.  For  material  and  submittal 
deficiencies,  contractors  were  often  late  in  submitting  material  submittals, 
and  they  also  commonly  ordered  materials  that  did  not  coincide  with  the 
original  submittal.  However,  most  projects  maintained  an  incident  rate 
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of  five  or  less,  with  three  projects  being  above  that,  and  one  as  high  as  24. 
The  material  submittal  incidents  were  slightly  more  normally  distributed 
(90  percent  between  0  and  20),  and  the  highest  count  was  28  incidents. 

Of  the  eight  quality  factors,  four  had  significant  variance.  The  most  common 
quality  problem  was  Electrical  work  (both  high  and  low  voltage;  M  =  4.0, 
SD  =  6.72).  The  project  with  the  most  Electrical  problems  had  28  recorded 
incidents.  Structural  issues  were  reported  second  most  commonly  ( M  =  3.0, 
SD  =  4.85).  The  projects  with  the  most  Structural  issues  had  14  and  20  inci¬ 
dents  respectively.  Most  projects  did  not  have  many  Building  Foundation 
problems  (M  =  2.7,  SD  =  5.8),  but  two  projects  had  12  and  28  each.  Lastly, 
Utility  issues  (M  =  1.7,  SD  =  2.72)  had  two  outliers  with  8  and  11  incidents. 
A  summary  of  project  performance  is  provided  in  Table  5,  and  an  accompa¬ 
nying  histogram  is  shown  in  Figure  2. 


TABLE  5.  PROJECT  DEFICIENCY  SUMMARY  j 

Deficiencies  (No.  of  Occurrences) 

Mean 

Median 

Standard 

Deviation 

Project  Management 

1.37 

0 

2.06 

Contract  Management 

1.70 

0 

2.75 

Design  Performance 

6.52 

5 

6.51 

Material  &  Submittals 

4.07 

2 

4.95 

Safety  Deficiencies 

2.56 

1 

5.22 

Reportable  Safety  Incidents 

0.76 

0 

1.33 

Horizontal  Work 

0.78 

0 

1.90 

Building  Foundation 

2.70 

1 

5.77 

Electrical 

4.00 

1 

6.72 

Mechanical 

0.52 

0 

0.90 

Utility 

1.74 

1 

2.72 

Structural 

3.00 

1 

4.85 

Interior  Finishing 

0.85 

0 

1.37 

Exterior  Finishing 

0.48 

0 

0.56 
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FIGURE  2.  CONSTRUCT  DEFECTS 


Results 

The  study  used  the  Mann-Whitney  pairwise  comparison  test  with  a 
2-sided,  normal  approximation  to  test  the  research  questions.  The  results, 
shown  in  Table  6,  indicate  that  there  are  five  significant  factors  and  one 
near-significant  factor  that  displayed  differences  across  contract  types.  The 
C/value  is  the  rank  assigned  to  the  variable;  the  z  is  the  test  statistic  value; 
and  the  “Sig.  (2-tailed)”  is  thep-valuefor  the  test.  Factors  were  determined 
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to  be  significant  if  they  possessed  ap-value  of  0.05  or  less.  The  Final  Cost, 
Awarded  Cost  Growth,  Final  Period  of  Performance,  Design  Performance, 
and  Contract  Management  were  significant  as  a  result  of  contract  type. 


TABLE  6.  MANN-WHITNEY  TEST  FOR  CONTRACT  TYPES  f 

Factor 

Type 

Mean 

Standard 

Deviation 

S 

Z 

Prob 

>|Z| 

Award  Amount 

CPFF 

FFP 

$25.6  M 
$25.3  M 

$17.6  M 
$26.4  M 

125 

-0.96 

0.338 

Final  Cost 

CPFF 

FFP 

$37.5  M 
$27.7 

$28.6  M 
$29.1  M 

105 

-2.05 

0.040* 

Awarded  Cost 

CPFF 

1.48 

0.38 

98 

-2.44 

0.015* 

Growth 

FFP 

1.13 

0.17 

Number  of  Contract 
Modifications 

CPFF 

FFP 

10.1 

7.4 

4.8 

1.8 

124 

-1.02 

0.308 

Change  Orders 

CPFF 

FFP 

3.4 

2.1 

2.8 

1.4 

120 

-1.26 

0.208 

Number  of  FPOP' 
Extensions 

CPFF 

FFP 

5.4 

3.3 

2.4 

1.3 

103.5 

-2.17 

0.030* 

Total  Days  Added  to 
Contract 

CPFF 

FFP 

591 

330 

275 

151 

97 

-2.49 

0.013* 

Initial  Period  of 
Performance 

CPFF 

FFP 

390 

373 

145 

145 

138.5 

-0.22 

0.827 

Final  Period  of 

CPFF 

945 

400 

105 

-2.05 

Performance 

FFP 

668 

209 

0.040* 

Awarded  Schedule 

CPFF 

2.46 

0.77 

107 

-1.94 

0.0522 

Growth 

FFP 

1.86 

0.61 

Security  Incidents 

CPFF 

FFP 

4.1 

6.9 

4.5 

7.3 

157 

0.74 

0.457 

Other  External 

CPFF 

5.4 

8.9 

151.5 

0.45 

0.653 

Environmental  Issues 

FFP 

8.2 

14.4 

Weather 

CPFF 

FFP 

22.9 

17.1 

22.4 

2.1 

126.5 

-0.88 

0.380 

Project  Management 

CPFF 

FFP 

1.3 

1.6 

2.1 

2.2 

154 

0.63 

0.526 

Contract 

CPFF 

1.2 

2.8 

180.5 

2.15 

0.031* 

Management 

FFP 

2.6 

2.7 

Design  Performance 

CPFF 

FFP 

3.9 

10.1 

2.7 

8.2 

187 

2.39 

0.017* 

Material  & 

CPFF 

6.1 

8.5 

119 

-1.32 

0.186 

Submittals 

FFP 

2.3 

3.7 
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TABLE  6,  CONTINUED 


Factor 

Type 

Mean 

Standard 

Deviation 

S 

Z 

Prob 

>|Z| 

Safety  Deficiencies 

CPFF 

FFP 

1.9 

3.9 

3.4 

7.0 

157.5 

0.80 

0.425 

Reportable  Safety 
Incidents 

CPFF 

FFP 

0.7 

0.8 

1.0 

1.7 

135 

-0.48 

0.631 

Horizontal  Work 

CPFF 

FFP 

0.6 

1.2 

0.9 

2.8 

140 

-0.17 

0.868 

Building  Foundation 

CPFF 

FFP 

2.1 

4.0 

3.2 

8.1 

151.5 

0.45 

0.652 

Electrical 

CPFF 

FFP 

3.4 

5.0 

4.6 

9.1 

132.5 

-0.58 

0.561 

Mechanical 

CPFF 

FFP 

0.5 

0.6 

1.2 

1.2 

152.5 

0.62 

0.531 

Utility 

CPFF 

FFP 

1.7 

2.1 

2.9 

2.6 

150 

0.37 

0.709 

Structural 

CPFF 

FFP 

3.2 

3.4 

5.3 

4.5 

149.5 

0.33 

0.738 

Interior  Finishing 

CPFF 

FFP 

0.6 

1.3 

1.0 

1.7 

156 

0.77 

0.438 

Exterior  finishing 

CPFF 

FFP 

0.6 

0.5 

1.2 

0.5 

151 

0.48 

0.628 

Note. 

’Signifies  2-tailed  significance  (p  <  0.05).  Reject  null  hypothesis. 

'FPOP  (Final  Period  of  Performance) 

^Nearly  significant;  and  is  significant  using  Fisher’s  Exact  Test  in  a  contingency  table. 


Final  Cost  for  wartime  projects  was  significantly  lower  for  FFP  contracts, 
as  suggested  by  the  literature.  Likewise,  Awarded  Cost  Growth  was  signifi¬ 
cantly  lower  for  FFP  contracts  as  well. 

Final  Period  of  Performance  was  lower  for  FFP  contracts.  The  Awarded 
Schedule  Growth  Index  was  calculated  by  dividing  the  final  government- al¬ 
lowed  period  of  performance  by  initial  contractual  period  of  performance 
(not  necessarily  the  actual  performance  period).  The  actual  period  of  per¬ 
formance  could  not  be  used  to  calculate  a  schedule  growth  factor  because 
of  the  inherent  differences  between  fixed-price  and  reimbursable  contracts. 
Fixed-price  projects  are  contractually  able  to  continue  in  operation  after 
the  contractual  completion  date  has  expired  because  the  contractor  is 
responsible  for  the  risk.  However,  reimbursable  contracts  must  be  closed 
out  when  the  period  of  performance  expires  unless  the  owner  extends  the 
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contractual  completion  date.  Therefore,  in  a  reimbursable  contract,  the 
actual  completion  date  is  always  the  same  or  before  the  contractual  date. 
Consequently,  actual  completion  dates  are  incomparable  between  the  con¬ 
tract  types.  Thus,  projects  were  compared  using  the  contractual  completion 
date.  Moreover,  the  contractual  completion  date  is  within  the  control  of  the 
owner  (whereas  actual  completion  in  fixed  contracts  is  not)  and  is  thereby 
a  superior  factor  to  compare  between  the  two  contract  types.  The  Awarded 
Schedule  Growth  was  a  near-significant  factor  in  the  Mann-Whitney  test. 
Therefore,  further  investigation  was  appropriate.  A  contingency  table  using 
Fisher’s  exact  test  revealed  that  Awarded  Schedule  Growth  depended  on 
contract  type  (p  =  0.0154). 

The  FFP  contracts  performed  worse  than  CPFF  contracts  in  terms  of 
Design  Performance  and  Contract  Management,  but  not  in  as-built  work. 
Contract  Management  was  defined  as  the  contractor’s  ability  to  fulfill  the 
administrative  requirements  of  the  contract.  Common  deficiencies  included 
missed  schedule  or  status  updates,  or  provision  of  adequate  living  and 
working  conditions  for  the  QA  engineers.  This  finding  suggests  that  while 
contractual  requirements  were  not  always  met  with  FFP  contracts,  the  fin¬ 
ished  facility  was  comparable  to  facilities  constructed  with  CPFF  contracts. 

Project  Management  deficiencies  were  a  separate  construct  than  Contract 
Management,  and  significant  differences  were  not  found  between  con¬ 
tract  types.  Our  definition  of  Project  Management  mirrors  closely  what 
Pinto  and  Winch  (2016)  describe  as  project  delivery  activities:  planning, 
execution,  controlling,  and  close-out.  Examples  of  Project  Management 
deficiencies  included  proceeding  with  work  without  approval  or  scheduling 
conflicting  craft  disciplines  in  the  same  work  area,  resulting  in  delays  and 
worker  conflicts.  No  Project  Management  differences  were  found  between 
contract  types. 

Design  Performance  was  carefully  defined  so  that  these  issues  did  not  over¬ 
lap  with  Project  or  Contract  Management.  Therefore,  these  issues  only 
included  design  quality  and  design  schedule  performance.  Although  the 
construction  agent  has  identified  several  design  flaws  postcontract  comple¬ 
tion,  no  occurrences  of  construction  failure  were  recorded  as  a  result  of  poor 
design.  The  most  frequently  observed  Design  Performance  deficiency  was 
late  design  submissions,  and  the  responses  to  these  issues  were  different 
across  contract  types.  These  late  submissions  caused  FFP  contractors  to 
work  at  risk.  Working  at  risk  occurs  when  designs  are  not  approved  by  the 
owner  and  the  contractor  decides  to  continue  with  construction,  knowing 
rework  may  occur  if  the  design  changes  before  it  is  approved.  This  rework 
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is  not  compensated  by  the  owner.  The  results  suggested  that  FFP  contrac¬ 
tors  were  willing  to  accept  this  risk  to  stay  on  schedule  and  to  avoid 
contractual  penalties.  On  the  other  hand,  the  CPFF  contractors  did  not  have 
as  many  instances  of  Design  Performance  deficiencies.  This  significant 
difference  suggests  that  the  contractors  were  likely  not  motivated  to  work 
at  risk  because  fixed  profits  were  guaranteed  and  they  did  not  fear  the 
accompanying  schedule  growth. 


Reimbursable  contracts  were  found  to  have  signifi¬ 
cantly  higher  costs  than  fixed  price  contracts.  This 
difference  was  found  for  cost  increases  during  the 
life  of  the  project  and  the  final  project  cost. 

Discussion 

Cost 

Reimbursable  contracts  were  found  to  have  significantly  higher  costs 
than  fixed  price  contracts.  This  difference  was  found  for  cost  increases 
during  the  life  of  the  project  and  the  final  project  cost.  Notably,  there  was 
not  a  significant  difference  in  Award  Amount  between  contract  types.  These 
findings  demonstrate  that  reimbursable  contracts  are  likely  to  be  awarded 
at  similar  prices  to  FFP  contracts,  but  are  likely  to  cost  more  at  the  end  of 
the  project.  The  validity  of  this  conclusion  is  strengthened  by  the  significant 
difference  seen  in  cost  growth.  In  the  analysis,  large  projects  were  compared 
alongside  small  projects;  and  there  may  have  been  considerable  variance 
between  the  project  factors,  which  may  reduce  the  credibility  of  a  direct 
comparison  in  terms  of  raw  cost  or  some  other  attribute.  The  Awarded  Cost 
Growth  Index  standardizes  the  projects’  cost  comparisons.  For  example, 
larger  projects  may  have  differences  in  risk  and  nature  of  work  than  smaller 
projects.  Additionally,  when  a  larger  project  experiences  delay,  it  ought  to 
cost  more  money  to  make  up  the  time  deficit.  The  Awarded  Cost  Growth 
Index  removes  unique  assignments  of  cost  to  enable  comparisons.  When 
this  was  done,  we  found  that  the  ratio  between  final  and  initial  costs  is  sig¬ 
nificantly  higher  for  reimbursable  contracts  versus  fixed-price  contracts. 
Higher  cost  growth  in  reimbursable  construction  contracts  aligns  with 
other  industry  research.  Reimbursable  contracts  do  not  incentivize  cost 
control  (Nkuah,  2006);  rather  they  may  incentivize  cost  growth  (Wamuziri, 
2013).  Thus,  as  expected,  wartime  construction  contracts  exhibited  the 
same  cost  behavior  as  peacetime  contracts. 
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Schedule 

The  average  time  required  to  complete  a  wartime  reimbursable  project 
is  greater  than  the  time  for  a  fixed-price  project.  This  is  consistent  with 
peacetime  findings  and  confirms  Jaszkowiak’s  (2012)  results  for  other 
Afghan  and  Iraq  U.S.  military  construction  projects.  The  observed  Awarded 
Schedule  Growth  is  expected  because  structurally  speaking,  schedule  and 
cost  growth  are  strongly  linked  in  reimbursable  contracts.  Whether  funding 
becomes  exhausted  due  to  slow  progress  or  unanticipated  cost  overruns, 
government  contracting  personnel  are  limited  in  their  options  for  reim¬ 
bursable  contracts  (assuming  all  costs  have  been  legitimized  during  invoice 
auditing).  To  continue  the  project,  they  must  provide  additional  funding, 
reduce  the  project  scope,  or  terminate  the  contract  in  its  current  state 
(FAR,  2015,  pts.  52.232-22).  Based  on  this  structural  connection,  we  would 
expect  contract  modifications  to  be  a  mediating  variable.  Indeed,  previous 
research  has  shown  that  contract  changes  are  closely  related  to  schedule 
performance  in  projects  (Ibbs,  2011).  While  total  number  of  Scope  Changes 
was  not  different  between  the  contract  types,  reimbursable  contracts  had 
more  schedule  modifications  than  fixed  contracts.  Additionally,  the  Total 
Days  Added  to  the  Contract  was  also  higher  for  reimbursable  contracts. 
Therefore,  the  results  suggest  that,  rather  than  Scope  Changes  being  the 
cause  of  Awarded  Schedule  Growth,  as  Ibbs  (2011)  suggested,  it  maybe 
some  other  mediating  factor  (or  possibly  the  contractor’s  lack  of  incentive 
to  adhere  to  the  schedule)  that  begets  more  Awarded  Schedule  Growth  in 
reimbursable  contracts. 
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Contract  types  also  had  a  near-significant  p -value  for  differences  in  the 
Awarded  Schedule  Growth  Index.  The  p-value  was  so  close  to  0.05  (unlike 
any  other  factor)  that  additional  analysis  was  performed  for  the  factor. 
A  contingency  table  showed  that  Awarded  Schedule  Growth  could  be 
dependent  on  contract  type.  Reimbursable  contracts  had  higher  Awarded 
Schedule  Growth  than  fixed  contracts.  This  reflects  similar  behavior  as 
discussed  with  Final  Cost:  contractors  for  reimbursable  contracts  may  not 
be  motivated  to  control  Awarded  Schedule  Growth  (Nkuah,  2006).  FFP 
contractors  are  incentivized  to  minimize  construction  costs  and  schedule, 
which  involves  indirect  costs  as  the  project  is  delayed.  CPFF  contractors 
do  not  have  these  inhibitions  for  either  cost  or  schedule.  The  construction 
agent  reported  that  contractors  would  often  divide  their  original  bid  by  the 
number  of  days  in  the  period  of  performance  to  establish  a  daily  burn  rate. 
Often,  the  daily  burn  rate  was  maintained  or  exceeded.  But  just  as  often, 
the  planned  schedule  was  not  met,  and  the  allocated  funds  were  exhausted 
before  the  project  was  complete.  Therefore,  when  more  time  was  granted 
to  the  project,  additional  funding  had  to  be  granted  to  complete  the  same 
project  (L.  Schoenenberger,  personal  communication,  2014).  By  design, 
CPFF  projects  have  greater  potential  for  Awarded  Schedule  Growth,  and 
this  research  found  that  for  this  sample,  on  average  they  did  exhibit  more 
Awarded  Schedule  Growth,  confirming  previous  literature. 

Quality 

Fixed-price  contracts  underperformed  compared  to  reimbursable  con¬ 
tracts  in  Design  Performance  and  Contract  Management.  The  daily  reports 
indicated  that  the  majority  of  the  reported  design  deficiencies  were  due  to 
incomplete  design  submissions  to  the  government.  The  incomplete  designs 
created  a  rework/resubmission  cycle.  The  contractors  would  choose  to  work 
at  risk  on  the  projects  (sometimes  for  months)— beginning  construction 
without  final,  approved  designs— in  order  to  meet  contractual  performance 
obligations.  Similarly,  the  contractors  frequently  worked  at  risk  as  they 
tried  to  comply  with  contract  management  tasks.  Contractors  would  miss 
submission  deadlines  and  would  have  difficulty  correcting  the  deficiency. 
However,  the  daily  reports  did  not  indicate  that  project  quality  was  directly 
affected  as  a  result  of  contractors  working  at  risk.  Acceptable  designs  or 
contract  submissions  were  eventually  submitted.  The  tests  suggest  that 
contractors  did  not  pay  as  close  attention  to  contract  and  design  documents 
on  fixed-price  contracts.  It  is  interesting  that  projects  were  able  to  continue 
successfully  in  spite  of  severely  late  design  submissions  and  approvals.  This 
may  confirm  previous  research  suggesting  there  are  unnecessary  steps  in 
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the  government  design-review  process,  or  that  some  details  of  design  are  not 
critical  to  project  completion  and  simpler  criteria  may  still  yield  a  successful 
project  (Blomberg,  Cotellesso,  Sitzabee,  &  Thai,  2013). 

This  study  found  no  significant  difference  in 
quality  performance  between  the  two  contract 
types.  This  conflicts  with  the  peacetime  expected 
outcome  of  quality  differences.  As  there  is  a  rela¬ 
tionship  between  time,  cost,  and  quality,  perhaps 
the  differences  are  manifest  only  in  the  observed 
time  and  cost  growth.  Our  results  also  contrast 
with  Jaszkowiak’s  (2012)  work.  Her  survey  of  con¬ 
struction  professionals  found  that  a  reimbursable 
project  tended  to  yield  better  quality  projects.  This 
research  did  not  find  any  craftsmanship  quality 
differences  between  fixed  price  and  reimbursable 
projects.  These  conflicting  results  maybe  attrib¬ 
utable  to  the  source  of  data.  Jaszkowiak  (2012) 
assessed  overall  perceptions  from  the  government 
construction  management  teams,  whereas  this 
study’s  data  consist  of  QA  deficiency  reports.  This 
research  did  not  analyze  customer  satisfaction  of 
the  project,  which  is  a  large  consideration  in  deter¬ 
mining  the  final  quality  of  a  project  (Baccarini, 

1999;  Lim  and  Mohamed,  1999).  Notwithstanding, 
this  research  suggests  that  heightened  deficiencies 
or  poor  quality  work  should  not  be  a  unique  subject 
of  focus  for  either  contract  type. 

Security  and  the  Environment 

Reimbursable  contracts  are  used  in  Afghanistan  by  the  U.S.  Government 
because  of  the  increased  risk  due  to  the  security  situation.  As  a  result,  it  was 
expected  that  external  environmental  factors  would  be  more  prevalent  on 
reimbursable  contracts.  The  use  of  this  contract  type  is  justified  because  of 
the  more  austere  or  uncertain  project  environments.  However,  there  was 
no  significant  difference  in  delays  due  to  any  of  the  external  environmental 
factors.  In  fact,  security  incidents  and  other  external  environmental  delays 
(e.g.,  local  interference)  were  reported  more  often  in  fixed-price  contracts 
though  not  significantly.  This  result  may  suggest  that  risk  assessments 
may  not  adequately  assess  the  security  situation  for  both  reimbursable  and 
fixed-price  projects.  Additionally,  the  term  ‘high  risk’  has  a  broad  meaning. 
A  project  may  have  been  high  risk  simply  due  to  being  in  a  remote  location  or 
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due  to  the  security  situation.  Additionally,  some  accessible  projects  are  clas¬ 
sified  as  high  risk  because  of  the  undefined  scope,  or  anticipation  of  many 
change  orders  as  the  end-user  firmed  up  requirements  (L.  Schoenenberger, 
personal  communication,  2014).  As  the  external  environment  was  not  a  sig¬ 
nificant  factor  between  contract  types,  these  findings  may  also  suggest  that 
the  high-risk  projects  are  characterized  more  by  vague  project  requirements 
than  by  the  environment. 


Limitations  and  Conclusions 

Limitations 

This  study  was  limited  to  25  projects,  which  restricted  the  statistical 
tests  to  nonparametr ic  tests  for  the  analysis.  Future  research  should  obtain 
a  larger  sample  group,  which  will  increase  the  number  of  analysis  options. 
Another  limitation  was  the  depth  of  data  retrieval  from  the  daily  reports. 
The  combined  length  of  the  daily  reports  was  approximately  20,000  pages. 
Therefore,  only  major  deficiencies  were  analyzed.  However,  there  were  many 
other  minor  incidents  recorded  by  the  QA  engineers.  In-depth  case  study 
research  on  smaller  groups  of  these  projects  may  provide  further  insight 
into  performance  differences  between  contracts. 

Conclusions 

The  purpose  of  this  research  is  to  provide  construction  agents,  firms, 
and  military  leaders  alike  with  information  that  will  aid  strategic  decisions 
regarding  future  military  construction  and  nation-building  projects.  All 
of  these  facts  underline  the  rapidly  changing  environment  that  is  wartime 
construction,  which  has  a  significant  effect  on  the  progress  of  a  project. 
The  results  largely  confirm  that  which  has  been  known  for  decades.  FFP 
contracts  achieve  lower  cost  and  schedule  growth  than  CPFF  contracts. 
Additionally,  we  found  similar  external  risk  profiles  for  both  types  of  con¬ 
tracts.  Both  contract  types  faced  similar  austere  conditions  in  terms  of 
physical  attacks  and  a  harsh  environment.  Nevertheless,  it  would  be  irre¬ 
sponsible  to  assume  that  FFP  contracts  are  more  advantageous  for  the 
government  to  use  in  a  wartime  environment.  There  were  specific  reasons, 
usually  risk-oriented,  that  led  the  construction  agent  to  use  CPFF  contracts, 
especially  in  the  initial  stages  of  the  Afghanistan  reconstruction.  Arguably, 
the  use  of  CPFF  may  have  prevented  the  default  of  contractors  on  more  high- 
risk  projects.  Instead,  the  message  of  this  article  is  that  owners  need  to  be 
aware  that  reimbursable  projects  are  likely  to  have  more  cost  and  schedule 
growth.  Owners  and  their  agents  need  to  take  proactive  steps  to  minimize 
the  growth  and  to  reduce  construction  inefficiencies. 
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Review: 

Richard  Whittle,  a  global  fellow  at  the  Woodrow  Wilson  International 
Center  for  Scholars,  longtime  military  journalist,  and  author  of  The  Dream 
Machine:  The  Untold  History  of  the  Notorious  V-22  Osprey  (reviewed  in 
the  Defense  ARJ,  Vol.  74,  July  2015)  continues  his  deep-dive  investigation 
of  high-profile  weapon  programs  by  this  time  unveiling  the  development 
of  the  Predator  drone.  Whittle  has  prepared  the  volume  drawing  on  hun¬ 
dreds  of  interviews  with  program  stakeholders,  and  5  years  of  (obviously 
careful)  research  that  eventually  granted  the  author  access  to  a  myriad 
of  supporting  documents.  Readers  of  the  Dream  Machine,  accustomed 
to  Richard  Whittle’s  methodology  and  style,  won’t  be  disoriented  by  this 
new  opus.  Predator’s  narrative  structure  is  essentially  a  reiteration  of  the 
previous  V-22  saga:  the  author  seeks  to  relate  the  individual  fates,  fortu¬ 
nate  or  unfortunate  political  decisions,  military  events,  and  operational 
anecdotes  that  shape  the  course  of  the  MALE  UAV  (Medium-Altitude  Long- 
Endurance  Unmanned  Aerial  Vehicle)  history,  from  its  inception  in  the  70s, 
to  Predator’s  armed  debut  after  the  9/11  attacks. 

In  the  introductory  chapter,  we  learn  that  one  of  the  most  effective  weap¬ 
ons  in  the  current  U.S.  arsenal  finds  its  origin  in  the  aftermath  of  the  Yom 
Kippur  war,  when  avisionary  Israeli  engineer— Abraham  Karem— pioneered 
the  deployment  of  unmanned  aircraft  to  collect  and  dispatch  real-time  tac¬ 
tical  information  on  enemy  positions.  For  acquisition  students,  the  second 
(and  central)  part  of  the  book,  is  undoubtedly  the  most  interesting  element 
of  Whittle’s  examination.  It  describes  how  long-endurance  UAVs  first  envi¬ 
sioned  in  the  80s  as  small,  cheap  observation  tools,  progressively  turned  into 
large  and  deadly  platforms  during  the  following  decade.  With  a  wealth  of 
detail,  the  author  recounts  the  technological  hurdles  drone  supporters  had 
to  overcome  during  this  20-year  development  marathon  (e.g.,  circumventing 
the  issue  of  remotely  piloted  operations),  as  well  as  the  evolution  of  military 
thinking  and  requirements  that  eventually  led  the  Air  Force  leadership  to 
weaponize  the  vehicle. 

The  book  is  superbly  researched,  well-structured,  and  easy  to  read.  Whittle 
has  an  unquestionable  talent  for  capturing  his  audience’s  attention  through 
a  compelling  and  thrilling  story-telling.  Readers  less  familiar  with  UAV 
jargon  and  airborne  technology  will  certainly  appreciate  the  effort  put  forth 
by  the  author  into  carefully  explaining  each  key  technological  development 
(e.g.,  the  installation  and  functioning  of  Hellfire  payload)  in  a  clear  and 
intelligible  way. 
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In  sum,  this  volume  is  a  worthwhile  read  for  anyone  interested  in  better 
comprehending  the  development  of  Predator,  and  the  subsequent  mass 
adoption  of  MALE  UAVs.  However,  beyond  the  usual  “bureaucratic  road¬ 
block”  and  “inter-Service  rivalry”  arguments,  it  only  adds  marginally  to 
our  understanding  of  the  weapon  systems  acquisition  process.  Fair  to  say. 
Whittle  shows  here  no  intention  to  deviate  into  this  type  of  analysis,  but 
as  such,  his  last  opus  might  nonetheless  present  a  more  limited  interest  for 
those  seeking  to  extrapolate  broader  conclusions  on  how  DoD  and  its  armed 
forces  procure  the  weapons  they  need. 
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Contributors  are  encouraged  to  seek  the  advice  of  a  reference  librarian  in 
completing  citation  of  government  documents  because  standard  formulas 
of  citations  may  provide  incomplete  information  in  reference  to  govern¬ 
ment  works.  Helpful  guidance  is  also  available  in  The  Complete  Guide  to 
Citing  Government  Documents  (Revised  Edition):  A  Manual  for  Writers 
and  Librarians  (Garner  &  Smith,  1993),  Bethesda,  MD:  Congressional 
Information  Service. 

Pages  should  be  double-spaced  and  organized  in  the  following  order:  title 
page  (titles,  12  words  or  less),  abstract  (150  words  or  less  to  conform  with 
formatting  and  layout  requirements  of  the  publication),  two-line  summary, 
list  of  keywords  (five  words  or  less),  body  of  the  paper,  reference  list  (only 
include  works  cited  in  the  paper),  author’s  note  or  acknowledgments  (if 
applicable),  and  figures  or  tables  (if  any). 

Figures  or  tables  should  not  be  inserted  or  embedded  into  the  text,  but  seg¬ 
regated  (one  to  a  page)  at  the  end  of  the  text.  When  material  is  submitted 
electronically,  each  figure  or  table  should  be  saved  to  a  separate,  exportable 
file  (i.e.,  a  readable  EPS  file).  For  additional  information  on  the  preparation 
of  figures  or  tables,  refer  to  the  Scientific  Illustration  Committee,  1988, 
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Biology  Editors,  Inc.  Restructure  briefing  charts  and  slides  to  look  similar 
to  those  in  previous  issues  of  the  Defense  ARJ. 

The  author  (or  corresponding  author  in  cases  of  multiple  authors)  should 
attach  a  signed  cover  letter  to  the  manuscript  that  provides  all  of  the  authors’ 
names,  mailing  and  e-mail  addresses,  as  well  as  telephone  and  fax  numbers. 
The  letter  should  verify  that  the  submission  is  an  original  product  of  the 
author(s);  that  all  the  named  authors  materially  contributed  to  the  research 
and  writing  of  the  paper;  that  the  submission  has  not  been  previously  pub¬ 
lished  in  another  journal  (monographs  and  conference  proceedings  serve  as 
exceptions  to  this  policy  and  are  eligible  for  consideration  for  publication  in 
the  Defense  ARJ);  and  that  it  is  not  under  consideration  by  another  journal 
for  publication.  Details  about  the  manuscript  should  also  be  included  in  the 
cover  letter:  for  example,  title,  word  length,  a  description  of  the  computer 
application  programs,  and  file  names  used  on  enclosed  DVD/CDs,  e-mail 
attachments,  or  other  electronic  media. 


COPYRIGHT 

The  Defense  ARJ  is  a  publication  of  the  United  States  Government  and 
as  such  is  not  copyrighted.  Because  the  Defense  ARJ  is  posted  as  a  complete 
document  on  the  DAU  homepage,  we  will  not  accept  copyrighted  manu¬ 
scripts  that  require  special  posting  requirements  or  restrictions.  If  we  do 
publish  your  copyrighted  article,  we  will  print  only  the  usual  caveats.  The 
work  of  federal  employees  undertaken  as  part  of  their  official  duties  is  not 
subject  to  copyright  except  in  rare  cases. 

Web-only  publications  will  be  held  to  the  same  high  standards  and  scru¬ 
tiny  as  articles  that  appear  in  the  printed  version  of  the  journal  and  will  be 
posted  to  the  DAU  Web  site  at  www.dau.mil. 

In  citing  the  work  of  others,  please  be  precise  when  following  the  author- 
date-page  number  format.  It  is  the  contributor’s  responsibility  to  obtain 
permission  from  a  copyright  holder  if  the  proposed  use  exceeds  the  fair  use 
provisions  of  the  law  (see  U.S.  Government  Printing  Office,  1994,  Circular 
92:  Copyright  Law  of  the  United  States  of  America,  p.  15,  Washington,  DC). 
Contributors  will  be  required  to  submit  a  copy  of  the  writer’s  permission  to 
the  managing  editor  before  publication. 

We  reserve  the  right  to  decline  any  article  that  fails  to  meet  the 
following  copyright  requirements: 
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•  The  author  cannot  obtain  permission  to  use  previously  copy¬ 
righted  material  (e.g.,  graphs  or  illustrations)  in  the  article. 

•  The  author  will  not  allow  DAU  to  post  the  article  in  our  Defense 
ARJ  issue  on  our  Internet  homepage. 

•  The  author  requires  that  usual  copyright  notices  be  posted 
with  the  article. 

•  To  publish  the  article  requires  copyright  payment  by  the 
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•  Author  checklist 

•  Biographical  sketch  for  each  author  (70  words  or  less) 

•  Headshot  for  each  author  should  be  saved  to  a  CD-R  disk  or 
e-mailed  at  300  dpi  (dots  per  inch)  or  as  a  high-print  quality 
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low-resolution  images  from  Web,  Microsoft  PowerPoint,  or 
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°  Title  (12  words  or  less) 

°  Abstract  of  article  (150  words  or  less) 

°  Two-line  summary 
°  Keywords  (5  words  or  less) 

°  Document  excluding  abstract  and  references  (4,500  words 

or  less  for  the  printed  edition  and  10,000  words  or  less  for 
the  online-only  content) 
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the  Defense  ARJ  Managing  Editor  at:  DefenseARJ@dau.mil. 
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experts  for  the  2016  Defense  Acquisition  Research  Jour¬ 
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•  Share  your  acquisition  research  results  with  the  Acquisition,  Technology,  and  Logistics 
(AT&L)  community. 
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•  Be  asked  to  speak  at  a  conference 
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We  welcome  submissions  from  anyone  in¬ 
volved  with  or  interested  in  the  defense  ac¬ 
quisition  process— the  conceptualization, 
initiation,  design,  testing,  contracting,  pro¬ 
duction,  deployment,  logistics  support,  mod¬ 
ification,  and  disposal  of  weapons  and  other 
systems,  supplies,  or  services  (including  con¬ 
struction)  needed  by  the  DoD,  or  intended  for 
use  to  support  military  missions. 


If  you  are  interested,  contact  the  Defense  ARJ  managing  editor  (DefenseARJ@dau.mil)  and 
provide  contact  information  and  a  brief  description  of  your  article.  Please  visit  the  Defense  ARJ 
Guidelines  for  Contributors  at  http://www.dau.mil/pubscats/Pages/ARJ.aspx, 
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the  Executive  Editor,  Defense  ARJ. 
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